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1 .  INTRODUCTION 


The  Federal  Aviation  Administration  (FAA)  Aviation  Weather  Development  Program,  with  support 
from  Massachusetts  Institute  of  Technology  Lincoln  Laboratory,  is  developing  the  Integrated  Terminal 
Weather  System  (ITWS)  to  support  safety  and  traffic  management  at  terminal  facilities.  The  ITWS  will  pro¬ 
duce  a  fully  automated,  integrated  terminal  weather  information  system  to  improve  the  safety,  efficiency  and 
capacity  of  terminal  area  aviation  operations.  The  ITWS  utilizes  data  from  FAA  terminal-area  weather 
systems  {e.g..  Terminal  Doppler  Weather  Radar,  Airport  Surveillance  Radar-9,  Low  Level  Windshear  Alert 
System),  National  Weather  Service  (NWS)  systems  {e.g..  Next  Generation  Weather  Radar  or  NEXRAD)  and 
other  systems  (e.g..  National  Meteorological  Center,  National  Lightning  Data  Network)  and  commercial  air¬ 
craft  to  create  its  products.  Safety  products  include  the  identification  of  hazardous  storms,  wind  shear 
(microbursts,  gust  fronts,  vertical  shears).  Traffic  management  products  include  three-dimensional  gridded 
winds  for  use  by  terminal  automation  systems,  approach  and  departure  corridor  winds,  and  storm  movement. 
The  ITWS  will  be  deployed  at  approximately  45  airports  starting  in  2000.  A  functional  prototype  began 
operating  in  1992  to  obtain  data  for  algorithm  development  and  to  demonstrate  ITWS  in  an  operational 
environment. 

The  ITWS  will  provide  for  more  efficient  planning  of  aircraft  movements  in  the  terminal/TRACON 
area  by  significantly  improving  the  quality  and  timeliness  of  near-term  predictions  of  weather  impacting  the 
local  area.  Identification  of  weather  impacts  specific  to  approach  and  departure  corridors,  comerposts,  run¬ 
ways,  and  the  airport  surface  will  enable  more  efficient  coordination  of  routing  strategies.  Aviators,  dispatch¬ 
ers,  traffic  managers,  controllers  and  airport  operations  managers  will  be  able  to  anticipate  rather  than  just 
react  to  these  weather  impacts.  Coordination  of  the  movement  of  traffic  through  alternate  arrival/departure 
routes  will  result  in  overall  increases  in  capacity.  The  ability  to  anticipate  impacts  (such  as  the  cessation  of 
significant  weather  in  the  area)  and  select  optimal  routes  or  holding  strategies  prior  to  the  arrival  of  traffic 
in  the  area  will  result  in  savings  of  time  and  fuel. 


1.1  1994  DEMONSTRATION  AND  VALIDATION 

During  the  summer  of  1994,  ITWS  functional  prototype  displays  (Figure  1)  were  deployed  at  the 
Memphis  Airport  Traffic  Control  Tower  (ATCT)  and  TRACON  (Terminal  Radar  Approach  Control),  at  the 
Memphis  Air  Route  Traffic  Control  Center  (ARTCC),  at  the  Orlando  International  ATCT  and  TRACON,  and 
at  the  Jacksonville  ARTCC.  (The  Sun  workstation  or  Situation  Display  is  used  to  present  graphical  weather 
data  to  traffic  managers.  The  Ribbon  Display  Terminal  is  used  to  display  wind  shear  and  microburst  alert 
messages  that  are  read  by  the  controller  directly  to  the  pilot.)  In  addition,  ITWS  displays  were  available  to 
the  NWS  forecast  offices  at  Memphis,  TN  and  Melbourne,  FL,  to  Northwest  Airlines  in  Minneapolis,  MN, 
and  to  Delta  Airlines  in  Orlando,  FL.  A  principal  objective  of  the  ITWS  testing  was  to  obtain  feedback  from 
operational  users  on  the  utility  of  ITWS  products  for  enhancing  safety  and  reducing  delays  and  controller 
workload.  Weather  products  were  provided  in  real  time  in  order  to  assess  the  utility  of  ITWS  products.  ITWS 
operations  at  these  facilities  provided: 


1 


3 


Figure  I .  The  ITWS  Situation  Display  and  Ribbon  Display  Terminal 


1 .  operational  demonstration  of  the  Initial  Operational  Capability  ITWS  products  as  part  of  the 
FA  A  Technical  Center’s  formal  Demonstration  and  Validation  (DemVal)  as  part  of  the  ITWS 
Operational  Test  and  Evaluation  program; 

2.  data  on  meteorological  phenomena  necessary  for  continued  refinement  of  the  ITWS  product 
generation  algorithms  prior  to  national  deployment;  and 

3.  operational  data  on  the  benefits  of  the  ITWS  products  in  meeting  the  terminal  weather 
information  needs  of  supervisors,  traffic  managers,  controllers,  pilots  and  Central  Weather 
Service  Unit  meteorologists. 

The  products  provided  to  the  users  included: 

1.  ITWS  Precipitation  with  anomalous  propagation  clutter  removed  (maximum  range  is  50 
nm) 

2.  ASR-9  Precipitation  with  anomalous  propagation  flagged 

3.  Storm  Motion 

4.  Storm  Extrapolated  Position 

5.  Storm  Cell  Information 

6.  Microburst  Detection  and  Prediction 

7.  Gust  Front  Detection,  Forecast,  and  Windshift  Estimate 

8.  Terminal  Winds 

9.  Terminal  Weather  Text  Message 

10.  Microburst  and  Wind  Shear  Countdown  Timers 

1 1 .  Lightning  Detection 

12.  Tornado  Detection  (pass  through  from  NEXRAD) 
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13.  Long  Range  Precipitation  (pass  through  from  NEXRAD;  maximum  range  is  200  nm) 

14.  Long  Range  Storm  Motion 

1.2  DEMVAL  AT  MEMPHIS  INTERNATIONAL  AIRPORT 


The  ITWS  DemVal  in  Memphis  began  on  23  May  and  ended  22  July  1994.  Regularly  scheduled  opera¬ 
tions  were  conducted  from  12:00  noon  to  7:00  PM  local  time  Monday  through  Friday,  for  a  total  of  about 
310  hours.  In  addition,  special  operations  were  conducted  to  support  air  traffic  operations  when  significant 
weather  occurred  outside  regularly  scheduled  operations.  For  example,  ITWS  products  were  provided  to  us¬ 
ers  before  noon  and/or  after  7  PM,  during  the  Federal  Express  pushes  (around  midnight  and  2  AM),  and  on 
weekends.  These  accounted  for  about  225  hours  of  additional  operations. 


The  microburst  and  wind  shear  products  were  based  on  Terminal  Doppler  Weather  Radar  (TDWR) 
data.  The  TDWR  radar  was  not  operational  during  the  first  two  days  of  DemVal  and  from  13  July  1994  until 
the  end  of  DemVal.  The  ITWS  continued  to  generate  and  deliver  the  remaining  products  during  these  time. 


1.3  DEMVAL  AT  ORLANDO  INTERNATIONAL  AIRPORT 


The  ITWS  DemVal  in  Orlando  began  on  11  July  and  ended  19  August  1994.  Regularly  scheduled  op¬ 
erations  were  conducted  from  12:00  noon  to  7:00  PM  local  time  seven  days  per  week,  for  a  total  of  280  hours. 
As  in  Memphis,  special  operations  were  conducted  when  significant  weather  occurred  outside  regularly 
scheduled  operations  to  support  air  traffic  operations.  These  accounted  for  about  10  hours  of  additional  opera¬ 
tions. 


1.4  ORGANIZATION  OF  THE  REPORT 


This  report  presents  technical  performance  assessments  of  the  ITWS  products.  Each  chapter  contains 
a  brief  description  of  the  product,  a  description  of  the  performance  assessment,  and  performance  results.  The 
performance  of  the  algorithms  is  measured  against  the  Minimum  Operational  Performance  Requirements 
(MOPR)  shown  in  TABLE  1 .  The  MOPR  describes  the  product  performance  necessary  for  Air  Traffic  Con¬ 
trol  to  accept  the  products  as  operationally  useful.  The  goals  specified  in  TABLE  1  indicate  the  desired  per- 
forrnance  for  the  products. 
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TABLE  1. 

Minimum  and  Desired  Operational  Performance  Requirements  for 
initial  Operational  Capability  iTWS  Products 


Parameter 

Threshold 

Goal 

Microburst  Prediction 

Probability  of  false  alert 

<  0.10 

<  0.05 

Prediction  lead  time 

<  2  minutes  prior  to  onset  of 
microburst  for  60%  of  predicted 
valid  wet  microburst  events 

<  2  minutes  prior  to  onset  of  the 
divergent  wind  shear  for  90%  of 
the  predicted  valid  wet 
microburst  events 

Gust  Front  Prediction 

Predicted  position  times 

Position  predicted  1 0  minutes 
and  20  minutes  in  advance 

Position  predicted  1 0  minutes 
and  20  minutes  in  advance 

Predicted  position  accuracy 

Predict  70%  of  gust  fronts 
impacting  airport  with  wind 
change  of  >  15  knots  10 
minutes  in  advance 

Predict  90%  of  gust  fronts 
impacting  airport  with  wind 
change  of  >  1 5  knots  1 0 
minutes  in  advance 

Probability  of  false  prediction 

Probability  of  false  10-minute 
prediction  <  0.10  for  gust  fronts 
with  wind  change  of  >  1 5  knots 

Probability  of  false  1 0-minute 
prediction  <  0.10  for  gust  fronts 
with  wind  change  of  >  15  knots 

Wind  shift  speed  accuracy 

maximum  of  ±  5  knots  or  10% 
of  speed  estimate  for  80%  of 
predictions  impacting  airport 

Wind  shift  direction  accuracy 

±  30  degrees  for  80%  of 
predictions  impacting  airport 

Storm  Motion 

Storm  speed  accuracy 

±  5  knots  for  90%  of  storms 
moving  at  >  1 0  knots 

±  5  knots  for  90%  of  storms 
moving  at  >  5  knots 

Storm  direction 

±  20  degrees  for  90%  of  storms 
moving  at  >  1 0  knots 

±  20  degrees  for  90%  of  storms 
moving  at  >  5  knots  and  ±  10 
degrees  for  50%  of  storms 
moving  at  >  5  knots 

Storm  Extrapoiated  Position 

Extrapolated  position  times 

Position  projected  10  minutes 
and  20  minutes  into  the  future 

Position  projeoted  10  minutes 
and  20  minutes  into  the  future 

Extrapolated  position 
accuracy 

1 0-minute  extrapolation  within  2 
nm  for  80%  of  storms  moving  at 
speeds  >  10  knots,  excluding 
those  storms  exhibiting 
significant  ( >  2  levels)  growth  or 
decay 

20-minute  extrapolation  within  2 
nm  for  70%  of  storms  moving  at 
speeds  >  1 0  knots,  including 
those  storms  exhibiting 
significant  growth  or  decay 
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TABLE  1.  continued. 


Storm  Cell  Information 

Storm  Cell  Association 

>  90%  of  features  associated  to 
correct  cell 

>  95%  of  features  associated  to 
correct  cell 

Storm  Cell  Information 

identify  80%  of  cells  which  will 
grow  or  decay  by  over  20%  in 
area  in  next  20  minutes 

ASR-9  AP  Edit 

Inadvertent  edit 

<  maximum  of  1 0  km^  of  or 

1 0%  of  90%  of  contiguous  areas 
with  weather  reflectivity  >  level 

3 

<  maximum  of  1 0  km^  of  or 

1 0%  of  90%  of  contiguous  areas 
with  weather  reflectivity  >  level 

2 

AP  editing  performance 

edit  70%  of  AP  when  ASR-9  AP 
is  >  level  3  and  is  at  least  2 
levels  above  actual  weather 
reflectivity  and  spatial  extent  of 
AP>  25km2 

Edit  85%  of  AP  when  ASR-9  AP 
is  >  level  3  and  is  at  least  2 
levels  above  actual  weather 
reflectivity  and  spatial  extent  of 
AP>  25  km2 

Terminal  Winds 

Horizontal  Resolution 

5  nm  out  to  30  nm  beyond 
TRACON  boundary  and  below 
23,000  ft 

1  nm  within  the  TRACON 
boundaries  and  below  18,000  ft; 

5  nm  elsewhere 

Vertical  Resolution  (betwreen 
levels) 

50  millibars 

25  mb  below  5000  ft  AGL  and 
within  1 5  nm  of  the  TDWR  radar; 
50  mb  elsewhere 

Accuracy 

±  1 0  knots  80%  of  time  in 
regions  and  at  times  when  both 
TDWR  and  NEXRAD  have  valid 
velocity  data 

±  5  knots  90%  of  time  in  regions 
and  at  times  when  both  TDWR 
and  NEXRAD  have  valid  velocity 
data 
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2.  ITWS  PRECIPITATION  AND  ASR-9  PRECIPITATION 

PRODUCTS 


The  Airport  Surveillance  Radar  (ASR)-9  radar  is  used  in  the  terminal  area  to  control  aircraft.  This  radar 
has  a  weather  channel  that  provides  the  location  and  intensity  of  precipitation  on  the  air  traffic  controllers’ 
radar  screen  (Weber,  1986).  TRACON  controllers  use  the  weather  information  to  aid  aircraft  in  avoiding 
weather.  The  ASR-9  radar  data  are  often  contaminated  by  ground  clutter  due  to  anomalous  propagation  (AP). 
Due  to  the  smoothing  process  used  in  the  ASR-9,  controllers  are  unable  to  distinguish  between  AP  and  valid 
weather  returns.  As  a  result  controllers  may  attempt  to  vector  aircraft  around  AP,  resulting  in  increased 
controller  workload  and  decreased  terminal  airspace  capacity. 


The  ITWS  product  suite  includes  two  precipitation  products:  ITWS  Precipitation  (AP  removed)  and 
the  ASR-9  Precipitation  (AP  flagged  in  black).  The  basis  for  these  products  is  the  ASR-9  weather  channel 
output.  Both  of  these  products  are  created  by  an  algorithm  called  AP-edit,  which  is  described  in  Klingle-Wil- 
son  et  al.,  1995. 


The  ITWS  Precipitation  product  is  a  representation  of  the  location  and  intensity  of  weather  in  the 
TRACON  area.  This  product  may  be  used  for  situational  awareness  and  as  a  planning  aid  for  air  traffic 
managers  by  showing  where  weather  is  located  relative  to  traffic  flow  patterns.  An  example  of  the  product 
is  provided  in  Figure  2. 


The  ASR-9  Precipitation  product  explicitly  shows  where  AP  is  located  relative  to  any  ASR-9  radar. 
Since  the  ITWS  Precipitation  product  does  not  replace  the  ASR-9  weather  display  on  any  controllers’  dis¬ 
plays,  the  Air  Traffic  Control  (ATC)  supervisor  or  traffic  manager  may  use  the  AP  flagged  product  to  deter¬ 
mine  the  location  of  AP  which  may  be  appearing  on  the  TRACON  controllers’  displays.  An  example  of  this 
product  is  provided  in  Figure  3. 


2.1  REMOVING  AP  CLUTTER  FOR  ASR-9  DATA 


The  ASR-9  weather  channel  provides  information  on  the  location  and  reflectivity  of  precipitation  in 
the  TRACON  area.  The  energy  emitted  by  the  radar  reflects  off  of  raindrops  in  the  atmosphere.  The  amount 
of  energy  reflected  indicates  the  size  and  number  of  the  raindrops  present;  heavy  rain  is  associated  with  higher 
intensity  levels.  The  returned  signal  is  passed  through  a  filter  that  removes  ground  clutter.  The  data  are 
smoothed  in  time  over  six  antenna  rotations  which  results  in  a  30-second  update  rate  of  the  precipitation  data. 
The  data  are  also  smoothed  in  space  to  a  resolution  of  0.5  nm.  During  this  smoothing  process,  the  spatial 
extent  of  the  highest  intensity  levels  becomes  exaggerated.  The  output  is  delivered  in  the  standard  NWS  VIP 
(Video  Integrator  and  Processor)  6-level  intensity  scale. 


The  data  from  the  ASR-9  weather  channel  may  be  contaminated  by  ground  clutter  due  to  AP.  In  the 
standard  atmosphere,  a  radar  beam  typically  travels  in  a  slightly  curved  path  whose  radius  of  curvature  is 
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Figure  3.  Example  of  the  ASR-9  Precipitation  product.  The  six  levels  are  color-coded  according  to  the  key  at  the  upper  right  of  the  figure.  Greens  represent  lighter 
precipitation;  reds  represent  heavier  precipitation;  black  represents  anomalous  propagation  (AP). 


greater  than  the  earth’s  radius.  Under  superrefraction  and  ducting  conditions,  the  path  of  the  beam  is  more 
highly  curved  toward  the  ground.  Energy  is  diverted  toward  the  ground  and  targets  that  would  normally  be 
below  the  radar  horizon  are  illuminated.  These  ground  clutter  returns  are  often  referred  to  as  anomalous 
propagation  clutter  or  AP  clutter.  Because  of  the  spatial  and  temporal  smoothing  performed  by  the  ASR— 9 
weather  channel  processing,  it  is  difficult  for  a  user  to  look  at  a  display  and  distinguish  AP  clutter  from  real 
weather  signals. 


The  atmospheric  conditions  that  cause  AP  are  temperature  inversions  and  moisture  gradients.  In  a  stan¬ 
dard  atmosphere,  temperature  decreases  with  height.  Sometimes  on  a  clear  night,  surface  cooling  reverses 
the  temperature  profile  such  that  temperature  increases  with  height  near  the  ground.  In  this  situation,  the 
ASR-9  radar  beam  is  bent  downward  and  strikes  the  ground.  The  ground  returns  look  like  real  weather  on 
the  ASR-9  display.  Although  the  skies  may  be  cloud-free,  this  “nocturnal  inversion”  causes  what  seems  to 
be  weather  returns  to  appear  on  the  ASR-9  displays.  As  the  inversion  strengthens  throughout  the  night,  the 
AP  increases  in  areal  extent  and  intensity.  This  condition  often  causes  the  AP  that  users  see  in  the  late  night 
to  early  morning  hours  (Figure  4a). 


In  addition  to  the  nocturnal  inversion  situation,  the  passage  of  a  cold  thunderstorm  outflow  over  or  near 
the  ASR-9  site  sets  up  an  inversion  condition  (cold  air  near  the  ground,  warm  air  aloft),  causing  AP.  In  this 
case,  valid  weather  returns  co-exist  with,  and  may  even  be  contaminated  by,  AP  returns.  In  the  latter  case, 
the  intensity  of  real  weather  appears  to  be  greater  than  it  actually  is  (Figure  4b). 


Weber,  et.al{\992>)  described  a  method  for  filtering  AP  from  an  the  ASR-Wind  Shear  Processor,  which 
is  an  ASR-9  radar  that  is  specially  configured  to  detect  wind  shear.  This  technique  makes  use  of  the  fact  that 
the  Doppler  velocity  of  AP  clutter  is  nearly  zero  and  that  the  spectrum  width  is  very  narrow.  Unfortunately, 
the  signal  processing  techniques  developed  for  that  system  require  access  to  the  Doppler  data.  For 
operational  ASR-9  systems  at  the  ITWS  sites,  the  ITWS  will  not  have  access  to  the  Doppler  data,  only  to 
the  6-level  smoothed  data.  Therefore,  the  approached  described  by  Weber  cannot  be  applied  to  operational 
ASR-9  radars  as  they  are  currently  configured  so  a  new  AP-editing  technique  using  data  from  the  NEXRAD 
pencil-beam  weather  radar  was  developed. 


An  AP  mask  is  created  that  contains  information  on  the  location  of  AP  and  the  maximum  allowable 
intensity  (VIP  level  1  through  6).  The  creation  of  the  AP  mask  is  initiated  by  the  receipt  of  the  NEXRAD 
Composite  Maximum  Reflectivity  (comprefl)  product.  When  the  comprefl  is  received,  it  is  first  searched  to 
determine  if  any  storms  are  present  {i.e. ,  an  attempt  is  made  to  identify  the  clear  sky,  nocturnal  inversion  case). 
If  the  search  cannot  identify  significant  weather  in  the  comprefl,  a  mask  is  created  which,  when  applied  to 
the  ASR-9  data,  causes  all  returns  to  be  removed.  All  returns  are  removed  fi-om  subsequent  ASR-9  updates 
until  a  new  comprefl  is  received  and  a  new  mask  is  created  (Klingle— Wilson,  et  al.,  1995) 


If  the  search  of  the  comprefl  indicates  that  weather  reflectivity  is  present,  a  pixel-by-pixel  (a  pixel  is 
0.5  nm  X  0.5  nm)  comparison  is  performed  to  determine  if  the  ASR-9  data  are  consistent  with  the  NEXRAD 
data.  ASR-9  data  collected  at  the  middle  of  the  NEXRAD  volume  scan  are  compared  to  the  comprefl  to  create 
the  AP  mask. 
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The  AP  mask  is  used  to  edit  the  ASR-9  scan  received  immediately  after  the  comprefl  product  and  all 
subsequent  ASR-9  scans  until  the  receipt  of  a  new  comprefl.  Only  those  grid  points  that  are  identified  as 
containing  AP  in  the  original  comparison  are  edited.  Depending  on  the  NEXRAD  volume  scan  being  used, 
the  AP  mask  could  as  much  as  nine  minutes  older  than  the  ASR-9  data  being  edited. 


Since  weather  moves  (especially  relative  to  the  AP  mask)  the  possibility  exists  that  real  weather  will 
propagate  into  an  area  that  was  previously  occupied  by  AP.  Unless  corrective  action  is  taken,  the  valid  weather 
returns  would  be  edited  in  accordance  with  the  AP  mask.  To  account  for  motion,  each  edited  ASR-9  scan 
is  passed  to  a  routine  that  finds  cells  of  at  least  VIP  level  3.  Motion  is  estimated  by  a  correlation  tracking 
technique  (Chomoboy,  1992;  Chomoboy  and  Matlin,  1994)  and  assigned  to  each  cell.  From  this,  the  time 
at  which  real  weather  is  expected  to  overlap  AP  regions  is  computed.  When  the  overlap  time  has  expired, 
the  pixel  in  the  AP  mask  is  “turned  off’  such  that  the  ASR-9  return  is  passed  through  without  change. 


2.2  TECHNICAL  PERFORMANCE  ASSESSMENT  AND  RESULTS 


There  are  basically  three  cases  that  must  be  considered  when  assessing  the  technical  performance  of 
the  AP-edit  algorithm:  all  AP,  all  weather,  and  mixed  AP  and  weather.  In  the  first  case,  all  returns  in  the 
ASR-9  data  are  AP  {i.e.,  the  clear  sky,  nocturnal  inversion  case).  In  the  second  case,  all  of  the  returns  in  the 
ASR-9  data  are  from  real  weather.  In  the  third  (and  most  stressful  for  the  algorithm),  AP  and  weather  returns 
co-exist;  AP  can  be  unassociated  with  any  valid  weather  returns  (referred  to  as  “clear  air  AP”)  and  weather 
returns  may  be  anomalously  high  due  to  contamination  by  AP.  Although  the  performance  metrics  for  each 
of  the  cases  are  the  same,  the  performance  of  the  algorithm  with  respect  to  these  cases  is  generally  considered 
independently  in  order  to  highlight  the  conditions  under  which  the  algorithm  might  fail. 


Truth  is  generated  by  a  human  expert  who  compares  raw  ASR-9  data  to  NEXRAD  data.  An  ASR-9 
datum  must  be  at  least  VIP  level  2  to  be  scored.  (According  to  the  TTWS  Users’  Working  Group  conducted 
in  November  1993,  level  1  precipitation  is  not  considered  operationally  significant.  In  addition,  the 
NEXRAD  radar  is  so  sensitive  that  level  1  AP  is  often  “confirmed”  by  NEXRAD  clear  air  returns.)  If  the 
NEXRAD  data  contain  no  areas  greater  than  5  nm^  of  valid  weather  returns  greater  than  17  dBZ,  a  flag  is 
set  such  that  the  automatic  scoring  software  recognizes  that  all  ASR-9  returns  should  be  removed. 


If  weather  is  present  in  the  NEXRAD  data,  visual  comparisons  of  the  ASR-9  and  NEXRAD  data 
(before  and  after  the  ASR-9  data)  are  performed.  Any  returns  in  the  ASR-9  data  that  do  not  correspond  to 
valid  weather  returns  in  the  NEXRAD  data,  either  in  location  or  intensity,  are  assumed  to  be  AP.  Polygons 
are  drawn  around  those  areas.  These  polygons  are  entered  into  a  file,  along  with  flags  indicating: 


a.  that  the  truther  is  unsure  if  the  polygon  contains  AP, 

b.  that  the  polygon  contains  AP  but  the  truther  is  unsure  of  the  correct  value  of  the 
weather,  or 
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c.  that  the  polygon  contains  AP  and  what  the  correct  weather  level  should  be. 


Truth  is  generated  for  ASR-9  scans  that  are  separated  in  time  by  about  90  seconds  (about  every  third  ASR-9 
update). 


The  performance  metrics  for  the  AP-edit  algorithm  are  PEAP  (probability  of  editing  AP)  and  PEW 
(probability  of  editing  weather).  PEAP  is  given  by  the  number  of  AP  pixels  correctly  edited  divided  by  the 
total  number  of  AP  pixels.  PEW  is  given  by  the  number  of  weather  weather  regions  incorrectly  edited  divided 
by  the  total  number  of  weather  regions,  so  the  goal  is  a  high  PEAP  and  a  low  PEW.  The  MOPR  for  the  AP-edit 
algorithm  are: 

1 .  <  maximum  of  1 0  km^  or  1 0  percent  of  90  percent  of  contiguous  areas  with  weath¬ 

er  reflectivity  >  level  3.  That  is,  no  more  than  the  greater  of  2.9  nm^  (10  km^)  or 
1 0  percent  of  the  contiguous  areas  of  >  VIP  level  3  weather  will  be  edited  (t.e.,  PEW 
<  0.10). 


2.  edit  70  percent  {i.e.,  PEAP  >  0.70)  of  AP  when  ASR-9  AP  is  at  least  level  3  and 
is  two  levels  above  actual  weather  reflectivity,  and  the  spatial  extent  of  AP  exceeds 
13.5  nm^  (25  km^). 


Table  2  provides  the  results  of  the  performance  analysis  for  nine  days  in  Memphis  (six  hours  from  three 
all-AP  days,  eight  hours  from  three  all-weather  days,  and  four  hours  from  three  mixed  days)  and  seven  days 
in  Orlando  (seven  hours  from  three  all-weather  days,  two  and  a  half  hours  from  three  mixed  days,  and  40 
minutes  from  one  all-AP  day.).  The  overall  performance  for  these  cases  significantly  exceeds  the  MOPR. 


One  would  expect  that  the  all-AP  days  would  have  a  PEAP  of  1 .0.  When  the  all-AP  condition  is 
identified,  the  ASR-9  scan  is  “wiped  clean”.  The  ability  to  identify  the  all-AP  condition  is  directly  tied  to 
the  quality  of  the  NEXRAD  data.  It  is  assumed  that  comprefl  is  free  of  ground  clutter  and  AP  contamination. 
Experience  in  Memphis  has  shown  that  this  is  seldom  the  case.  The  comprefl  on  a  clear-air  day  then  does 
not  pass  the  no-weather  test  and  the  AP-edit  algorithm  is  forced  into  the  more  conservative  pixel-by-pixel 
edit.  ASR-9  AP  can  be  confirmed  as  real  weather  based  on  NEXRAD  ground  clutter  and  AP,  with  a  resulting 
decrease  in  PEAP. 


The  AP  mask  is  created  at  the  NEXRAD  update  rate.  If  the  ASR-9  scan  used  to  create  the  AP  mask 
is  from  the  middle  of  the  NEXRAD  six-minute  volume  scan,  the  AP  mask  is  three  minutes  old  relative  to 
the  first  ASR-9  scan  to  which  it  is  applied  and  nine  minutes  older  than  the  last  application.  Evolution  of  AP 
(and  weather)  for  that  time  is  not  represented  in  the  AP  mask.  As  the  AP  mask  ages,  the  editing  performance 
AP  decreases.  This  is  exemplified  in  Figure  5. 
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Table  2. 

REAP  and  PEW  for  all-AP,  All-weather,  and  Mixed  AP-Weather  Cases. 


1 - 

All  AP 

All  Weather 

Mixed  1 

1  Memphis  | 

PEAP(>  level!) 

0.97 

- 

0.80 

PEAP(>  level  3) 

0.98 

- 

0.96 

PEW 

- 

0.00 

0.01 

Oris 

indo 

PEAP(>  level!) 

0.94 

- 

0.85 

PEAP(>  levels) 

0.97 

- 

0.76 

PEW 

- 

0.01 

0.02 

Both  Lc 

ications 

PEAP(>  level!) 

0.97 

0.81 

PEAP(>  levels) 

0.98 

- 

0.95 

PEW 

0.01 

0.02 

The  data  in  Figure  5  are  from  2300  to  2330  UT  17  July  1994  in  Memphis  (a  mixed  case).  The  vertical 
lines  indicate  the  times  of  new  AP  masks.  Generally,  the  highest  PEAP  values  are  found  when  the  AP  mask 
is  first  applied  to  the  ASR-9  data,  when  the  AP  mask  is  newest.  As  the  AP  mask  ages  relative  to  the  ASR-9 
data,  performance  drops  because  AP  continues  to  evolve,  making  the  AP  mask  less  representative  of  the 
situation.  This  trend  is  more  pronounced  when  the  level  2  AP  is  included  in  the  analysis.  It  is  interesting  to 
note  that  over  the  time  period,  overall  performance  improves  because  the  AP  intensified  but  did  not  develop 
beyond  the  original  AP  area. 


There  was  less  AP  in  Orlando  than  in  Memphis.  The  three  Orlando  mixed  cases  had  smaller  AP  areas 
than  the  Memphis  mixed  cases.  In  the  Memphis  mixed  cases,  the  total  number  of  AP  pixels  of  at  least  level 
3  was  on  the  order  of  2  x  10^;  for  at  least  level  2,  the  total  number  was  around  5  x  10^.  In  the  Orlando  mixed 
cases,  the  total  number  of  AP  pixels  of  at  least  level  3  was  on  the  order  of  1  x  10“^;  for  at  least  level  2,  the 
total  number  was  around  5x10“*.  The  PEAP  and  PEW  values  vary  from  case  to  case.  In  general,  the  algorithm 
performs  much  better  if  the  weather  or  AP  is  wide-spread. 
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Figure  5.  Time  series  ofPEAPfor  level  2  and  greater  (*)  and  level  3  and  greater  (+)  AP. 


The  approach  used  to  remove  AP  from  ASR-9  data  is  very  conservative  to  insure  that  real  weather 
returns  are  not  inadvertently  identified  as  AP  and  removed.  Nonetheless,  preliminary  scoring  results  indicate 
that  over  80  percent  of  AP  is  correctly  edited  from  the  ASR-9  data.  In  addition,  less  than  one  percent  of  con¬ 
tiguous  weather  regions  are  edited. 
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3 .  ITWS  STORM  CELL  INFORMATION  PRODUCT 


The  ITWS  Storm  Cell  Information  (SCI)  product  provides  a  textual  description  of  storm  attributes 
which  cannot  be  deduced  from  the  ITWS  Precipitation  product  alone.  This  text  describes  the  height  of  the 
storm  (echo  top),  whether  the  storm  cell  contains  hail  an^or  lightning,  and  whether  the  storm  contains  a  me- 
socyclone  (that  is,  a  strong  rotation  with  strong  updrafts).  The  term  “severe  storm  circulation”  is  used  for  a 
mesocyclone  detection.  This  product  uses  the  ITWS  precipitation  product  to  identify  storm  cells,  the  NWS 
NEXRAD  radar  for  identification  of  echo  tops,  hail,  and  mesocyclones,  and  the  National  Lightning  Detection 
Network  for  cloud-to-ground  lightning  detection.  SCI  is  used  for  situational  awareness  and  as  an  aid  to  plan¬ 
ning  traffic  flow  in  the  TRACON  area.  An  example  of  the  product  is  provided  in  Figure  6. 

3.1  STORM  CELL  INFORMATION  ALGORITHM 

The  generation  of  SCI  text  is  performed  in  three  basic  steps:  gridding  of  storm  information,  storm  cell 
detection,  and  text  generation  (Dasey,  et  al,  1995).  Mesocyclones  and  severe  hail  (greater  than  3/4-inch  di¬ 
ameter  at  the  ground)  detections  from  the  NWS  NEXRAD  radar  are  provided  as  point  locations.  Hail  detec¬ 
tions  are  converted  into  areas  (or  grids)  based  on  their  association  with  high  reflectivity  regions  identified 
from  the  comprefl.  Mesocyclone  detections  are  converted  to  grids  based  on  their  physical  size.  The  echo  tops 
from  NEXRAD  are  already  provided  in  grid  format.  Cloud-to-ground  lightning  stroke  detections  are  accu¬ 
mulated  into  a  gridded  image  which  depicts  a  two-minute  average  lightning  flashrate.  Each  of  these  grids 
is  reoriented  so  it  is  centered  on  the  airport  reference  point.  Furthermore,  these  grids  may  be  the  result  of  data 
which,  in  the  case  of  hail,  echo  tops  and  mesocyclones,  are  as  much  as  six  minutes  old.  Storm  motion  esti¬ 
mates  from  the  ITWS  Storm  Motion  product  are  used  to  compensate  these  gridded  data  for  motion  which 
occurred  over  the  age  of  the  detections. 

Running  in  parallel  to  this  gridding  process  is  an  algorithm  which  uses  the  ITWS  Precipitation  map 
to  create  outlines  of  precipitation  at  VIP  levels  3  through  6.  A  final  association  process  searches  the  area  with¬ 
in  an  outline,  which  defines  each  storm  cell  detection,  for  sufficiently  high  levels  of  the  other  hazards.  If  the 
probability  of  severe  hail,  or  if  lightning  flashrate  is  sufficient,  a  hail  or  lightning  text  message  is  added  to 
the  SCI  text  box.  If  a  mesocyclone  is  present,  this  is  indicated  as  a  “SEVERE  STORM  CIRCULATION”  in 
the  SCI  text  box.  Finally,  the  maximum  echo  top  within  the  cell  outline  is  reported.  SCI  information  is  deter¬ 
mined  within  the  highest  precipitation  level  of  a  storm.  If  that  precipitation  area  is  considered  too  small,  the 
cell  is  dilated  on  all  sides  repeatedly  until  a  minimum  area  threshold  is  met. 

Figure  7  shows  a  four  panel  display  illustrating  the  generation  of  the  product.  The  upper  left  panel  illus¬ 
trates  the  VIP  level  4  and  5  storm  cell  outlines  overlaid  on  the  ITWS  precipitation  grid  for  an  Orlando  storm. 
The  upper  right  panel  shows  how  these  outlines  correspond  to  the  grid  of  the  probability  of  severe  hail,  with 
the  lightning  flashrate  and  echo  top  grids  shown  in  the  lower  left  and  right,  respectively.  The  extreme  lower 
right  comer  displays  the  SCI  text  that  is  generated  for  the  VIP  level  5  outlines,  based  on  the  overlap  of  that 
outline  with  the  grids. 

3.2  TECHNICAL  PERFORMANCE  ASSESSMENT  AND  RESULTS 

The  measures  of  performance  for  the  SCI  product  are  simply  whether  or  not  the  reported  hazards  for 
a  given  storm  cell  match  those  in  the  local  area  surrounding  the  cell  as  they  were  reported  by  the  various  sen- 
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Figure  6.  Example  of  the  SCI  product.  The  text  box  at  the  lower  right  corner  of  the  figure  provides  information  about  the  storm  cell  to  which  the  black  line  is  drawn. 
HAIL  indicates  a  greater  than  50  percent  probability  of  the  presence  of  severe  hail  at  the  ground;  LIGHTNING  indicates  a  cloud-to-ground  lightning  flashrate  of 
three  flashes  per  minute  in  the  storm;  ECHO  TOP  provides  an  estimate  of  the  echo  top  of  the  storm  above  ground  level  in  units  of  (x  1 00)  feet;  SEVERE  STORM 
indicates  the  presence  of  sP'ongly  rotating  horizontal  winds  and  strong  updrafts  in  the  storm. 


Figure  7.  Example  of  how  the  SCI  product  is  generated.  Storm  cell  outlines  are  overlaid  on  (a)  the  ITWS  6-level  precipitation,  (b)  probability  of  severe  hail  (percent), 
(c )  lightning  flashrate  (flashes  per  minute;  blue  dots)  and  mesocyclone  detections  ( red  squares),  and  ( d)  echo  tops  (xlOOO  feet).  The  data  for  each  grid  are  searched 
within  the  outlines  to  determine  the  appropriate  information  to  issue  in  the  text  box. 


sors.  Correct  identification  of  the  hazards  associated  with  a  given  storm  cell  is  a  hit.  The  presence  of  a  particu¬ 
lar  hazard  in  the  vicinity  of  the  storm  cell  not  reported  by  the  SCI  is  a  miss.  The  association  of  a  hazard  to 
a  storm  cell  that  does  not  contain  that  hazard  is  a  false  alarm. 

Truth  for  the  SCI  product  consists  of  the  output  from  the  Storm  Stmctures  Algorithm  (SSA;  Eilts  et 
al,  1995)  package  (hail  and  mesocyclone  detections),  the  NEXRAD  Echo  Tops  product  used  in  the  associa¬ 
tion  of  echo  tops  to  cells,  and  lightning  detections.  Since  the  SSA  products  are  (or  will  be)  NEXRAD  prod¬ 
ucts,  it  is  assumed  that  these  inputs  will  be  validated  by  the  NEXRAD  Program  Support  Facility.  Lightning 
detections  are  validated  in  a  separate  analysis  in  Chapter  VIIL 

The  SCI  product  is  generated  by  overlaying  reflectivity  contours  (storm  cells)  on  the  various  data  fields 
and  searching  within  the  contour  to  determine  the  value  to  be  assigned  to  the  storm  cell.  These  values  are 
reported  in  the  SCI  text  box.  The  minimum  area  to  be  searched  is  42  nm^.  If  a  contour  does  not  encompass 
the  minimum  search  area,  it  is  dilated  by  the  grid  resolution  (0.5  nm)  on  all  sides  repeatedly  until  the  mini¬ 
mum  search  area  is  met. 

For  hail,  a  hit  is  declared  if  the  search  area  of  a  storm  cell  contains  a  Probability  of  Severe  Hail  (POSH) 
value  equaling  or  exceeding  a  threshold  (currently  60  percent)  and  “HAIL”  is  reported  in  the  SCI  text  box. 
A  miss  is  declared  if  the  search  area  of  a  storm  cell  contains  a  POSH  value  of  at  least  60  percent  and  “HAIL” 
is  not  reported  in  the  SCI  text  box.  In  addition,  a  miss  is  declared  if  a  POSH  value  of  at  least  60  percent  is 
not  associated  with  a  storm  cell.  A  false  alarm  is  declared  for  a  storm  cell  whose  SCI  text  box  contains  “HAIL” 
but  whose  search  area  does  not  contain  a  POSH  value  of  at  least  60  percent. 

Mesocyclone  detections  are  represented  by  a  bounding  boxes  whose  x,y  dimensions  are  equal  to  the 
radius  of  the  largest  two-dimensional  shear  feature.  A  hit  is  declared  if  the  search  area  of  a  storm  cell  overlaps 
any  part  of  a  mesocyclone  detection  and  the  SCI  text  box  contains  the  message  “SEVERE  STORM  CIR¬ 
CULATION”.  A  miss  is  declared  if  an  overlap  occurs  but  the  SCI  text  box  does  not  contain  “SEVERE 
STORM  CIRCULATION”  or  a  mesocyclone  bounding  box  is  not  associated  to  a  storm  cell.  A  false  alarm 
is  declared  for  a  storm  cell  whose  SCI  text  box  contains  “SEVERE  STORM  CIRCULATION”  but  whose 
search  area  does  not  overlap  a  mesocyclone  detection. 

For  lightning,  storm  cell  contours  are  overlaid  on  a  map  of  cloud-to-ground  lightning  flashes  that  oc¬ 
curred  within  the  two  minutes  prior  to  the  storm  cells.  A  hit  is  declared  if  the  storm  cell  search  area  contains 
at  least  one  lightning  flash  and  the  SCI  text  box  contains  the  message  “LIGHTNING”.  A  miss  is  declared 
if  a  search  area  contains  at  least  one  lightning  flash  but  the  SCI  text  box  does  not  contain  “LIGHTNING”. 
A  false  alarm  is  declared  for  a  storm  cell  whose  SCI  text  box  contains  “LIGHTNING”  but  whose  search  area 
does  not  contain  at  least  one  lightning  flash. 

For  echo  tops,  a  hit  is  declared  if  the  maximum  value  of  echo  top  that  is  found  within  the  search  area 
is  reported  in  the  SCI  text  box.  A  miss  is  declared  if  the  echo  top  value  reported  in  the  SCI  text  box  is  not 
the  maximum  found  within  the  search  area  or  if  there  is  no  echo  top  report  in  the  SCI  text  box  when  echo 
top  value  fall  within  the  search  area.  A  false  alarm  is  declared  when  the  reported  echo  top  exceeds  the  maxi¬ 
mum  value  falling  within  the  search  area  or  when  echo  top  is  not  reported. 
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The  MOPR  for  this  product  is  at  least  90  percent  correct  cell  association  unless  constrained  by  sensor 
input.  That  is,  90  percent  of  storm  phenomena  (hail,  lightning,  mesocyclones,  and  echo  tops)  will  be 
associated  to  the  correct  storm  cell  {i.e.,  regions  of  high  reflectivity).  The  results  of  the  analysis  for  six  days 
in  Memphis  and  two  days  in  Orlando  are  presented  in  TABLE  3.  Blank  entries  in  the  table  indicate  that  the 
hazard  was  not  present.  Over  all  days,  the  algorithm  exceeded  the  MOPR  for  all  hazards.  The  performance 
in  Memphis  and  Orlando  did  not  differ  greatly. 


TABLE  3. 

Performance  Statistics  for  the  Storm  Celi  information  Aigorithm. 


Date 

HAIL 

MESOCYCLONE 

LIGHTNING 

ECHO  TOP  1 

Memphis  | 

04/15/94 

- 

1.0(10/10) 

0.86  (398/461) 

1.0  (1015/1015) 

06/08/94 

- 

*- 

0.90  (388/429) 

1.0  (920/920) 

06/09/94 

0.90  (85/94) 

0.50  (5/10) 

0.97  (912/945) 

1.0  (1072/1075) 

06/16/94 

- 

- 

0.92  (499/540) 

1.0  (920/920) 

06/21/94 

0.97  (33/34) 

0.92  (524/568) 

1.0  (983/983) 

06/26/94 

0.98  (109/111) 

1.0(57/57) 

0.98  (208/212) 

1.0(221/221) 

Total 

0.95  (227/239) 

0.94  (72/77) 

0.93  (2929/3155) 

1.0  (5131/5134) 

Orlando  | 

07/12/94 

1.0  (5/5) 

- 

0.89  (614/692) 

1.0  (1020/1020) 

07/16/94 

0.93  (28/30) 

- 

0.93  (1251/1333) 

1.0(1723/1723) 

Total 

0.94  (33/35) 

- 

0.92  (1865/2025) 

1.0  (2743/2743) 

Both  1 

Total 

0.95  (259/274) 

0.94  (72/77) 

0.93  (4794/5180) 

1 .0  (7874/7877)  | 
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4.  ITWS  STORM  MOTION  AND  STORM  EXTRAPOLATED 

POSITION  PRODUCTS 


The  ITWS  Storm  Motion  product  provides  an  estimate  of  the  motion  (speed  and  direction)  of  storms 
in  the  terminal  area.  Motion  is  indicated  using  arrows  (for  direction)  and  alphanumerics  (for  speed).  When 
overlaid  on  the  ITWS  Precipitation  product.  Storm  Motion  may  be  used  as  a  plarming  aid  to  better  anticipate 
the  need  for  runway  and  airspace  configuration  changes.  An  example  of  the  Storm  Motion  product  overlaid 
on  the  ITWS  Precipitation  product  is  provided  in  Figure  8. 


The  Storm  Extrapolated  Position  (SEP)  product  should  be  viewed  as  a  supplement  to  the  Storm  Motion 
product.  It  provides  leading-edge  contours  of  cells  and/or  cell  groups  and  extrapolates  these  contours  to  indi¬ 
cate  the  likely  positions  of  these  cells  projected  10  and  20  minutes  into  the  future,  assisting  the  users  in  esti¬ 
mating  the  time  at  which  a  given  cell  will  impact  a  terminal  route  or  runway.  The  extrapolations  may  be  used 
to  estimate  the  impact  times  of  fast-moving  storms.  An  example  of  the  product  is  shown  in  Figure  9. 


4.1  STORM  MOTION/STORM  EXTRAPOLATED  POSITION  ALGORITHMS 


4.1.1  Using  ITWS  Precipitation 


The  Storm  Motion  product  uses  the  ASR-9  weather  channel  data  for  its  knowledge  of  current  precipita¬ 
tion  levels.  An  image  processing  technique  is  used  that  compares  two  precipitation  images  which  are  sepa¬ 
rated  in  time.  It  is  assumed  that  differences  between  the  two  images  result  solely  from  the  motion  of  the  weath¬ 
er;  storm  growth  and  decay  are  not  considered.  Briefly,  the  method  segments  a  radar  image  into  overlapping 
15  mn  X  15  nm  regions.  For  the  DemVal,  the  two  images  were  thresholded  at  NWS  level  3.  Spatial  correlation 
is  performed  between  the  weather  in  a  given  15  nm  x  15  nm  region  over  the  two  scans.  Figure  10  illustrates 
this  spatial  partitioning  and  full-grid  analysis. 


The  ability  of  the  technique  to  compute  motion  is  a  function  of  the  resolution  of  the  input  data  and  the 
speed  of  the  weather.  The  update  rate  of  the  ASR-9  precipitation  map  is  30  seconds,  but  storms  do  not  move 
far  enough  in  30  seconds  for  the  technique  to  compute  an  accurate  motion.  Hence,  Storm  Motion  estimates 
are  computed  by  comparing  images  that  are  nominally  separated  in  time  by  four  minutes.  This  sampling, 
coupled  with  the  map’s  spatial  resolution,  limits  the  accuracy  of  motion  vectors  to  ±3.8  knots  (90  percent 
confidence  interval;  Chomoboy  and  Matlin,  1994).  However,  this  can  be  mitigated  somewhat  by  taking  ad¬ 
vantage  of  the  ASR-9  high  update  rate. 


In  convective  situations,  storm  motion  can  represent  a  composite  of  actual  storm  movement  and  appar¬ 
ent  motion  due  to  growth  and  decay.  For  example.  Midwest  squall  lines  are  usually  observed  to  move  from 
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Figure  10.  Example  of  the  generation  of  the  Storm  Motion  product.  The  Storm  Motion  algorithm  takes  a 
15 nm  X 15  nm  area  {highlighted  region  in  a)  and  finds  the  best  match  for  it  in  a  scan  from  four  to  five  minutes 
earlier.  This  procedure  is  repeated  using  overlapping  regions  to  construct  a  grid  with  7.5  nm  spacing.  For 
external  display.  Storm  Motion  interpolates  the  grid  to  cells  as  illustrated  in  b. 


west  to  east.  Individual  storm  cells  move  to  the  northeast  witliin  the  squall  line,  with  new  cells  developing 
at  the  southern  end  of  the  squall  line  and  old  cells  dissipating  at  the  northern  end.  In  situations  ol  extended 
“steady-state”  growth  and  decay,  the  Storm  Motion  vector  could  indicate  movement  to  the  east-northeast. 
In  general,  however.  Storm  Motion  is  dominated  by  the  movement  of  the  individual  cells. 


Storm  Extrapolated  Position  interprets  the  internal  motion  grid  computed  by  the  Storm  Motion  algo¬ 
rithm  and,  like  Storm  Motion,  uses  the  ASR-9  weather  channel  data  for  its  knowledge  of  current  precipitation 
levels.  Tlie  SEP  algorithm  (Chornoboy  and  Matlin,  1994)  computes  storm  contours  using  NWS  level  3  as 
its  contouring  level.  Before  contouring,  however,  closely  associated  cells  are  grouped,  filling  in  and  smooth¬ 
ing  level  3  gaps  of  1  to  2  nm.  The  internal  Storm  Motion  grid  is  used  to  determine  leading  and  trailing  edges 
and  only  the  leading  edge  of  each  contour  is  displayed.  These  contours  are  extrapolated  10  and  20  minutes 
into  the  future.  The  algorithm  designates  a  certain  subset  of  SEP  contours  and  vectors  as  “default”  for  display. 

4.1.2  Using  NEXRAD  Data 


Beyond  the  range  of  foe  ASR-9  coverage,  NEXRAD  data  is  used  to  infer  motion.  Processing  is  analo¬ 
gous  to  ASR-9  usage  except  that  images  are  separated  in  time  by  the  NEXRAD  volume  scan  time  (six  to 
H)  minutes)  and  may  have  a  decreased  spatial  resolution.  Hence,  long-range  motion  requires  a  few  changes 
in  algorithm  parameters  to  accommodate  NEXRAD’s  data  resolution  and  update  rate.  Eor  DemVal.  a  2.2  nm 
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X  2.2  nm  NEXRAD  composite  product  (nominal  update  rate  of  six  minutes)  was  used.  The  resolution  limita¬ 
tions  for  motion  detection  requires  looking  across  NEXRAD  maps  separated  in  time  by  12  minutes.  Conse¬ 
quently,  there  is  a  reduction  in  the  expected  accuracy  for  motion  estimates  using  the  NEXRAD  data. 


4.2  TECHNICAL  PERFORMANCE  ASSESSMENT  AND  RESULTS 


The  MOPR  for  Storm  Motion  are: 

1 .  Storm  Speed  Accuracy:  ±  5  knots  for  90  percent  of  storms  moving  at  10  knots  or 
greater. 

2.  Storm  Direction  Accuracy:  ±20  degrees  for  90  percent  of  storms  moving  at  10 
knots  or  greater. 

The  Storm  Motion  algorithm  implemented  in  DemVal  was  changed  little  from  earlier  implementations 
and  as  such  can  be  expected  to  meet  the  performance  characteristics  documented  by  Chomoboy  (1991).  Al¬ 
though  that  report  focused  on  performance  using  TDWR  data,  it  can  be  taken  as  a  lower  bound  on  expected 
DemVal  performance,  as  discussed  by  Chomoboy  and  Matlin  (1994). 

The  validation  of  requirements  for  speed  accuracy  and  direction  accuracy  using  recorded  DemVal  data 
is  a  planned  exercise  which  is  currently  underway,  and  these  results  will  be  provided  in  a  future  report.  Briefly, 
human  scorers  will  work  a  set  of  DemVal  data  days  (selected  from  the  days  considered  below)  and  create  a 
tmth  data  base  that  represents  the  human’s  capacity  to  detect  and  measure  storm  motion.  This  truth  data  base 
will  be  characterized  for  its  variation  and  used  to  score  Storm  Motion  performance  accordingly. 

There  are  no  formal  performance  requirements  that  apply  to  the  long-range  motion  product,  which  uti¬ 
lizes  the  same  algorithm  albeit  with  a  NEXRAD  derived  data  source.  The  expected  degradation  in  perfor¬ 
mance  owing  to  decreased  sampling  resolution  and  update  rate  have  been  noted  above. 

The  MOPR  for  Storm  Extrapolated  Position  are: 

1 .  Extrapolated  Position  Times:  Position  projected  10  minutes  and  20  minutes  into  the  fu¬ 
ture 

2.  Extrapolated  Position  Accuracy:  10-minute  extrapolation  within  2  nm  for  80  per¬ 
cent  of  storms  moving  at  speeds  of  greater  than  10  knots,  without  growth  or  decay 
of  2  levels  or  greater 

During  DemVal,  the  10-  and  20-minute  extrapolated  positions  were  computed  and  displayed  as  re¬ 
quired  in  the  MOPR. 
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Appropriateness  of  the  smoothed  contour  representation  is  based  on  a  point-by-point  tally  of  contour 
coordinates  exported  for  external  display.  Truth  consists  of  unsmoothed  contours,  drawn  at  the  actual  storm 
level  boundary,  and  the  scoring  measures  the  distance  between  each  point  of  the  smoothed  contour  and  the 
truth  contour.  The  distance  between  a  point  and  a  given  contour  curve  is  the  smallest  distance  obtained  from 
point-to-point  comparison  of  the  reference  point  with  each  point  (coordinate)  in  the  contour  curve. 


Accuracy  of  extrapolated  position  contours  is  assessed  as  a  binary  statistic  applied  to  each  10-minute 
contour  exported  for  display.  That  is,  a  contour  segment  is  subjectively  scored  as  being  either  consistent  or 
not  consistent  with  its  accuracy  criterion.  A  tally  of  accurate  10-minute  contours  is  made.  Accuracy  at  the 
level  of  20-minute  extrapolation  is  the  same  as  that  of  10-minute  extrapolation  except  to  the  extent  growth 
and  decay  degrade  performance.  The  displacement  error  due  to  ITWS  computational  inaccuracies  for  the 
leading-edge  displacement  shall  be  less  than  1  mu  95  percent  of  the  time  for  a  10-minute  extrapolation. 


A  subjective  scoring  method  for  SEP  contours  has  been  described  by  Chomoboy  and  Matlin  (1994), 
and  that  report  also  includes  preliminary  results  from  1993  operations  in  Orlando,  FL  and  Dallas,  TX.  As 
detailed  below,  this  analysis  was  extended  to  form  a  preliminary  evaluation  of  DemVal  data. 


Storm  growth  and  decay  are  significant  factors  for  which  the  current  SEP  algorithm  does  not  account. 
Adherence  to  the  MOPR  is  conditioned  on  non-obscuration  by  observed  significant  growth  and/or  decay. 
Specifically,  SEP  contours  were  identified  as  belonging  to  one  of  three  categories:  Hit,  Miss  and  Miss  due 
to  Evolution  (significant  growth  and/or  decay).  A  hit  is  an  SEP  that  was  a  human  expert  has  determined  is 
within  1  nm  of  the  true  edge.  The  category  of  Miss/Evolution  represents  SEPs  that  could  not  be  scored  for 
accuracy  owing  to  ambiguity  introduced  by  growth  and  decay  (a  few  instances  arose  where  neither  growth 
nor  decay  could  be  cited  explicitly,  yet  the  scorer  felt  storm  evolution  invalidated  scoring).  To  their  best  abil¬ 
ity,  the  human  scorers  treated  10-  and  20-minute  evaluations  as  independent  (cross  referencing  was  not  used 
to  aid  interpretation).  All  Miss/Evolution  SEPs  are  excluded  when  considering  SEP  motion  accuracy.  The 
Miss  category  contains  all  those  situations  not  covered  by  the  first  two  and  are  considered  SEP  motion  fail¬ 
ures. 


Eleven  operational  days  were  selected  for  analysis,  with  every  attempt  made  to  obtain  a  uniform  selec¬ 
tion  across  site  (Memphis  vs  Orlando)  and  with  respect  to  average  storm  speed.  Most  storms  with  estimated 
speeds  greater  than  25  knots  are  from  the  Memphis  data  base. 


First,  two  issues  not  represented  in  the  above  data  set  or  summary  analysis  deserve  comment. 

1 .  During  the  very  first  days  of  Memphis  operation,  long  extended  line  storms  (not  ex¬ 
perienced  in  Orlando  in  1 993)  revealed  a  deficiency  in  the  sub-algorithm  that  gener¬ 
ates  leading-edge  contours.  One  day  quite  representative  of  this  problem  was  9  June 
1994  (Memphis).  The  algorithm  was  corrected  for  the  remaining  (major)  portion  of 
the  DemVal,  and  no  further  gross  violations  were  identified  during  the  demonstra¬ 
tion  period.  This  technical  issue  can  be  addressed  more  comprehensively  with  future 
upgrades,  and  these  initial  deficiencies  are  not  included  in  the  statistics  below. 
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2.  The  SEP  algorithm  implemented  for  DemVal  did  not  take  account  of  the  ASR  data 
horizon.  Consequently,  SEP  contours  were  created  along  the  data  boundary,  and  ex¬ 
trapolations  were  shown  beyond  the  range  of  the  ASR.  This  condition  was  not  con¬ 
sidered  serious  enough  to  warrant  correction  during  the  DemVal.  For  the  purpose 
of  scoring,  however,  these  cases  were  not  included  either  because  the  required 
correction  is  simple  and  straight  forward. 


The  validation  of  SEP  performance  consists  of  comparing  the  number  of  SEP  contours  marked  as  hits 
versus  those  maiked  as  misses.  The  Miss/Evolution  category  is  not  considered  in  regard  to  the  fulfillment 
of  MOPR  because  it  represents  instances  where  the  human  scorer  could  not  discern  the  correctness  of  the 
motion  information.  The  statistics  for  growth/decay  are  significant  in  documenting  the  severity  of  growth/ 
decay  effects  in  the  two  DemVal  environments. 


TABLE  4  contains  the  sum  tallies  for  SEP  scoring.  Adding  the  results  for  Memphis  and  Orlando,  there 
is  a  0.96  Hit  rate  for  10-minute  extrapolation  and  a  0.88  rate  for  20-minute  extrapolation. 


TABLE  4. 

SEP  Scoring  Statistics  by  Location 


Hits 

Miss 

Miss/Evoiution 

10  Minutes 

Memphis 

0.96 

(2997/3110) 

0.04 

(113/3110) 

740 

Oriando 

0.96 

(2659/2759) 

0.04 

(100/2759) 

571 

Both 

0.96 

(5656/5869) 

0.04 

(213/5869) 

1311 

20  Minutes 

Memphis 

0.85 

(2474/2909) 

0.15 

(435/2909) 

950 

Oriando 

0.92 

(2323/2529) 

0.08 

(206/2529) 

801 

Both 

0.88 

(4797/5438) 

0.12 

(641/5438) 

1751 

TABLE  5  re-examines  the  same  data  with  the  counts  stratified  to  illustrate  the  effects  of  average  storm 
speed  (average  of  all  Storm  Motion  velocities  for  the  day’s  operation).  Qearly,  the  10-minute  results  meet 
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the  MOPR.  For  this  preliminary  analysis,  no  effort  was  made  to  further  understand  the  natme  of  algorithm 
misses  in  the  Miss  category.  Because  the  computed  average  speeds  represent  an  average  speed  for  the  day/ 
storm,  TABLE  5  also  represents  a  stratification  by  day,  and  it  is  not  yet  clear  what  effect  individual  storms 
had  in  biasing  the  summed  results.  A  more  detailed  examination  therefore  appears  warranted. 


TABLE  5. 

SEP  Scoring  Statistics  by  Average  Storm  Speed  Excluding  Storms  Which  Exhibit 

Growth  and/or  Decay 


Hits 

Miss 

Miss/Evolution 

10  Minutes 

5-10  knots 

0.92 

(2342/2540) 

0.08 

(198/2540) 

414 

10-25  knots 

0.97 

(2420/2505) 

0.03 

(85/2505) 

762 

>25  knots 

0.96 

(1001/1040) 

0.04 

(39/1040) 

218 

All  speeds 

0.95 

(5163/6085) 

0.05 

(322/6085) 

1394 

20  Minutes 

5-10  knots 

0.81 

(2001/2471) 

0.19 

(470/2471) 

483 

10-25  knots 

0.97 

(1958/2029) 

0.03 

(71/2029) 

1023 

>25  knots 

0.85 

(838/986) 

0.15 

(148/986) 

272 

All  speeds 

0.87 

(4797/5486) 

0.13 

(689/5486) 

1778 

TABLE  6  presents  the  performance  for  all  storms.  When  storms  exhibiting  significant  growth  and/or 
decay  are  included,  only  77  percent  of  the  SEP  contours  provide  a  good  match  to  the  actual  precipitation  lead¬ 
ing  edges.  Only  66  percent  of  the  20-minute  extrapolations  are  a  good  approximation  to  the  actual  precipita¬ 
tion  leading  edge  20  minutes  in  the  future.  Since  ITWS  Air  Traffic  need  an  accurate  depiction  of  the  weather 
situation  in  the  future  to  make  timely  planning  decisions,  the  performance  of  the  20-minute  SEP  clearly  needs 
improvement  if  it  is  to  be  used  operationally  as  an  accurate  precipitation  future  position  prediction.  This  is 
a  principle  goal  of  the  ITWS  preplanned  product  improvement  work.  In  addition,  the  limitations  of  the  IOC 
SEP  product  need  to  be  addressed  explicitly  in  training  for  Air  Traffic  planners. 
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TABLE  6. 

SEP  Scoring  Statistics  by  Average  Storm  Speed  Including  Storms  Which  Exhibit  Growth 

and/or  Decay. 


Hits 

Miss 

10  Minutes 

5-10  knots 

0.79 

(2342/2954) 

0.21 

(612/2954) 

10-25  knots 

0.74 

(2420/3267) 

0.26 

(847/3267) 

>25  knots 

0.80 

(1001/1258) 

0.20 

(257/1258) 

All  speeds 

0.77 

(5763/7479) 

0.23 

(1716/7479) 

20  Minutes 

5-10  knots 

0.68 

(2001/2954) 

0.32 

(953/2954) 

10-25  knots 

0.64 

(1958/3052) 

0.37 

(1144/3052) 

>25  knots 

0.67 

(838/1258) 

0.33 

(420/1258) 

All  speeds 

0.66 

(4797/7264) 

0.34 

(2467/7264) 
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5.  ITWS  MICROBURST  PRODUCTS 


The  ITWS  Microburst  algorithms  are  responsible  for  the  detection  and  prediction  of  strong  divergent 
outflows  of  wind  near  the  ground  surface  generated  from  storm  downdrafts.  The  detection  and  prediction 
portions  of  the  product  are  actually  separate  algorithms  whose  outputs  are  combined  into  one  set  of  display 
shapes  and  text  messages.  The  purpose  of  this  product  is  to  enhance  safety  of  landing  and  departing  aircraft. 
The  graphical  alerts  (shapes)  are  displayed  on  the  Situation  Display  in  the  tower  and  TRACON.  The  text  mes¬ 
sages  are  displayed  on  the  Ribbon  Display  at  each  tower  controller  position.  These  messages  are  read  directly 
to  pilots  on  final  approach  or  departure.  An  example  of  the  product  is  provided  in  Figure  1 1 . 


5. 1  MICROBURST  DETECTION  ALGORITHM 


The  diverging  outflow  of  cold  air  resulting  from  a  strong  storm  downdraft  can  result  in  a  rapid  transition 
from  a  headwind  to  a  tailwind  for  an  aircraft  encountering  the  event  on  takeoff  or  landing.  The  increased  lift 
from  the  headwind  can  cause  an  unaware  pilot  to  reduce  speed  and  angle  the  aircraft  nose  downward,  which 
accentuates  the  decreased  lift  on  the  tailwind  side.  When  the  microburst  is  strong  and  the  aircraft  is  low,  even 
an  aware  pilot  may  not  be  able  to  maintain  safe  flight.  Most  existing  wind  shear  detection  systems,  such  as 
LLWAS  (Low  Level  Windshear  Alert  System)  and  TDWR,  warn  about  microbursts  by  specifying  the  antici¬ 
pated  change  in  wind  speed  (headwind  to  tailwind)  along  the  flight  path.  The  ITWS  microburst  detection 
algorithm  has  been  improved  to  take  advantage  of  recent  studies  by  National  Aeronautics  and  Space  Admin¬ 
istration  Langley  and  several  manufacturers  of  airborne  wind  shear  detection  systems.  Microburst  and  wind 
shear  severity  are  more  closely  related  to  the  rate  of  the  wind  speed  change  (shear),  rather  than  the  magnitude 
of  the  change  itself.  This  means  that  a  50-knot  wind  speed  loss  over  distance  of  2  nm  is  inherently  more  severe 
than  a  50— knot  wind  speed  loss  over  a  distance  of  4  nm.  The  use  of  shear,  rather  than  wind  speed  difference, 
results  in  warnings  which  more  closely  reflect  the  danger  to  the  aircraft. 


Each  surface  scan  by  the  TDWR  provides  a  picture  of  the  component  of  the  winds  flowing  toward  or 
away  from  the  radar.  The  ITWS  microburst  detection  algorithm  converts  these  Doppler  velocities  to  a  two-di¬ 
mensional  map  of  the  shear  along  the  path  of  the  radar  beams.  The  shear  map  is  searched  for  regions  of  strong 
diverging  shear  (indicating  winds  at  the  surface  are  spreading  -  a  key  clue  to  microburst  presence).  The  search 
is  performed  at  several  shear  levels,  and  the  areas  of  strong  shear  are  analyzed  for  peaks  to  attempt  to  create 
one  output  shape  per  microburst  event.  The  shapes  are  created  as  circles,  unless  the  algorithm  has  strong  evi¬ 
dence  that  the  microburst  event  is  not  completely  circular,  in  which  case  the  TDWR-like  bandaid  shapes  are 
output.  Finally,  the  alerts  are  checked  for  precipitation  on  the  ground  or  above  the  event  to  verify  that  enough 
moisture  is  present  to  cause  a  microburst.  This  helps  remove  false  alarms  created  by  other  phenomena  which 
may  look  like  microbursts  (such  as  a  flock  of  birds  taking  off  from  the  ground  in  all  directions). 


In  previous  experiments  with  the  TDWR  testing,  microburst  alerts  equaling  the  maximum  strength  of 
each  shape  impacting  the  runway  were  output.  Users  testing  the  system  remarked  about  overwaming  of  mi- 
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Figure  II.  Example  of  the  ITWS  microburst  detections  on  the  Situation  Display.  The  red  circular  regions  overlaying  the  precipitation  map  indicate  the  presence  of 
microbursts  and  wind  shear  events;  filled  shapes  correspond  to  microbursts,  hollow  shapes  correspond  to  wind  shears  even  ts.  The  messages  that  would  appear  on  the 
Ribbon  Display  are  shown  in  the  gray  panel  at  the  cen  ter  right  of  the  display. 


croburst  and  wind  shear  severities.  Since  that  time,  the  microburst  strength  is  downgraded  based  on  the  extent 
to  which  it  overlaps  the  runway  corridor.  When  an  alert  shape  intersects  a  runway  flight  path,  or  is  very  close 
to  it,  the  runway  alert  level  is  calculated  by  assuming  that  the  strongest  hazard  is  through  the  center  of  the 
event.  The  runway  velocity  change  is  then  calculated  by  dividing  the  maximum  length  of  the  overlap  of  the 
alert  with  the  runway  corridor  by  diameter  of  the  alert  shape  and  multiplying  by  the  maximum  alert  strength. 
This  is  depicted  in  Figure  12. 


_  1.0  nm 

Fractional  length  overlap  =  — — 

1.5  nm 

Flight  path  strength  =  (1.0/1.5)*40  kts  =  26.7  kts 

Flight  path  strength  =  25  kts  (rounded) 

Text  Message: 

18A  WSA  25K-  IMF 

(18  Approach,  Wind  Shear  Alert,  25  knot  loss  at  l-mile  final) 


18 


Figure  12.  Strategy  for  the  generation  of  a  microburst  alert.  The  diameter  of  the  40-knot  microburst  is 
1 .5  nm.  The  length  of  the  microburst  shape  that  lies  along  the  approach  corridor  is  1  nm. 


5.2  TECHNICAL  PERFORMANCE  ASSESSMENT  AND  RESULTS 


Flight  path  truth  data  were  generated  by  reviewing  the  TDWR  data  with  the  appropriate  runway  ARE¬ 
NA  (area  noted  for  attention)  overlay.  Doppler  radar  products  needed  to  generate  the  truth  include  radial  shear, 
radial  velocity,  and  VIL  (vertically  integrated  liquid  water).  An  event  is  considered  a  wind  shear  or  microburst 
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if  the  following  criteria  are  met.  The  event  must  contain  a  continuous  region  of  at  least  4.2  km/nm  (4  m/s/km) 
of  radial  shear  extending  several  radials  in  azimuth  and  at  least  0.27  nm  (0.5  km)  along  the  flight  path.  The 
maximum  velocity  difference  (delta V)  along  a  radial  within  the  event  must  be  at  least  1 5  knots  over  a  distance 
of  1.85  nm  (1  km)  or  less.  In  addition,  a  VIL  value  of  at  least  5  kilograms  per  square  meter  must  be  in  close 
proximity  (within  1  nm)  of  the  event.  Once  the  above  criteria  are  met,  the  maximum  deltaV  observed  within 
an  ARENA  is  recorded  as  the  strength  of  the  truth  event. 

The  performance  of  the  Microburst  Detection  algorithm  is  assessed  using  the  metrics  of  probability 
of  detection  (POD)  and  probability  of  false  alarm  (PFA).  Probability  of  detection  is  defined  as  the  number 
of  correct  detections  divided  by  the  number  of  true  events.  Probability  of  false  alarm  is  defined  as  the  number 
of  detections  not  supported  by  truth  divided  by  the  number  of  algorithm  alarms.  These  numbers  are  generated 
for  both  the  wind  shear  (greater  than  15  knots)  and  microburst  (greater  than  30  knots)  alert  levels.  Perfor¬ 
mance  is  based  on  runway  alerts  where  TDWR  can  substantiate  the  loss.  Overwaming  and  underwaming 
probabilities  also  are  computed.  Overwaming  occurs  when  microburst  alerts  are  incorrectly  issued  for  wind 
shear  events.  Underwaming  occurs  when  wind  shear  alerts  are  incorrectly  issued  for  microburst  events.  The 
scoring  is  performed  using  the  contingency  table  approach  and  the  tolerances  discussed  by  Cole  and  Todd, 
1993. 


The  results  from  the  1 994  DemVal  are  provided  in  TABLE  7.  These  statistics  are  for  five  days  in  Mem¬ 
phis  and  eight  days  in  Orlando.  From  these  tables,  the  POD,  PFA,  and  probabilities  of  overwaming  and  under¬ 
waming  are  computed.  These  performance  statistics  are  presented  in  TABLE  8. 


TABLE  7.  Contingency  Tables  for  Microburst  Algorithm  Performance. 


1  Memphis  | 

EVENT  1 

ALERT 

Null 

Wind  Shear 

Microburst 

Total 

Null 

0 

8 

0 

8 

Wind  Shear 

2 

163 

1 

166 

Microburst 

0 

1 

11 

12 

Total 

2 

172 

12 

186 

1  Orlando  | 

EVENT  1 

ALERT 

Null 

Wind  Shear 

Microburst 

Total 

Null 

0 

10 

0 

10 

Wind  Shear 

20 

550 

6 

576 

Microburst 

0 

23 

126 

149 

Total 

20 

583 

132 

735 
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TABLE  8. 

Performance  Statistics  for  the  Microburst  Detection  Algorithm. 


Memphis 

Orlando 

Both 

POD  (wind  shear) 

0.95  (163/172) 

0.94  (550/583) 

0.94  (713/755) 

POD  (microburst) 

0.92  (11/12) 

0.95  (126/132) 

0.95  (137/144) 

PFA  (wind  shear) 

0.01  (2/178) 

0.03  (20/725) 

0.02  (22/903) 

PFA  (microburst) 

0.00  (0/178) 

0.00  (0/725) 

0.00  (0/903) 

P(overwarn) 

0.08(1/12) 

0.15(23/149) 

0.15  (24/161) 

P(underwarn) 

0.01  (1/166) 

0.01  (6/576) 

0.01  (7/742) 

Wind  Loss  Estimates* 

0.90  (158/176) 

0.79  (555/703) 

0.81  (713/879) 

*  Wind  loss  estimates  within  5  knots  or  20  percent  of  the  actual  loss. 


5.3  MICROBURST  PREDICTION  ALGORITHM 


Microburst  outflows  form  as  a  thunderstorm  downdraft  reaches  the  ground  and  spreads  out.  Signs  of 
this  downdraft  can  be  identified  by  the  Microburst  Prediction  algorithm  using  TDWR  precipitation  data  and 
temperature  and  humidity  data  from  surface  observations,  NWS  models,  and  commercial  aircraft  (tempera¬ 
ture  only).  Microburst  predictions  provide  an  average  lead  time  of  two  minutes  for  developing  microburst 
events.  This  provides  a  “heads  up”  to  pilots  lining  up  for  final  approach  or  takeoff  and  compensates  for  any 
latency  in  the  Microburst  Detection  algorithm  (Wolfson,  etal.,  1994) 


For  each  microburst  prediction,  a  computer  process  checks  to  see  if  the  prediction  overlaps  either  a 
wind  shear  alert  (WSA)  or  a  microburst  alert  (MBA).  If  no  overlap  occurs,  a  15-knot  WSA  is  issued.  If  the 
prediction  does  overlap  a  WSA,  the  WSA  is  upgraded  to  a  30-knot  MBA.  Finally,  if  a  detected  MBA  is  al¬ 
ready  present,  no  change  is  made  to  the  display.  With  this  concept,  no  new  symbols  are  introduced  on  the 
Situation  Display  and,  if  the  predictions  impact  the  runway  corridors,  both  the  controllers  and  pilots  will  be 
made  aware  of  them  through  the  Ribbon  Display.  They  will  appear  as  actual  WSA  or  MBA  messages. 


5 .4  TECHNICAL  PERFORMANCE  ASSESSMENT  AND  RESULTS 


The  algorithm  attempts  to  predict  microbursts  which  are  (a)  between  2.7  and  16.2  nm  from  the  radar, 
and  (b)  at  least  three  degrees  from  the  edge  of  the  TDWR  scanning  sector.  Within  this  region  features  aloft 
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can  be  identified  and  followed  for  developing  microburst  parent  storms.  A  microburst  is  defined  as  a  diver¬ 
gent  wind  loss  event  which  attains  a  loss  of  over  30  knots.  A  wind  shear  is  defined  as  a  wind  loss  event  which 
attains  a  loss  of  15  knots.  Truth  consists  of  human-identified  microburst  events. 


The  performance  metrics  for  microburst  prediction  are  probability  of  predicting  an  event  (POP)  and 
probability  of  false  prediction  (PFP).  POP  is  the  likelihood  that  a  true  event  will  be  predicted.  PFP  is  the  li¬ 
kelihood  that  an  individual  prediction  does  not  match  (within  nine  minutes)  an  event  whose  strength  is  greater 
than  or  equal  to  15  knots. 

The  POP  is  determined  by  trying  to  match  each  predicted  shape  with  the  truth  event  most  closely 
associated  with  the  prediction.  The  association  is  accomplished  by  comparing  the  location  of  the  prediction 
relative  to  the  prospective  matching  event.  The  truther  searches  up  to  nine  minutes  forward  to  find  matching 
events.  A  prediction  can  only  match  one  microburst  event.  Generally  a  criteria  of  25  percent  overlap  of  the 
two  regions  is  required,  however  very  small  alerts  or  predictions  (less  than  0.5  nm  in  radius)  that  are  within 
0.5  nm  of  the  prediction  are  considered  hits.  POP  is  computed  as  the  number  of  correctly  predicted  events 
divided  by  the  total  number  of  predictable  events. 

The  PFP  is  calculated  using  the  minute-by-minute  predictions.  All  predictions  that  do  not  match  an 
event  are  considered  false.  The  PFP  is  computed  as  the  number  of  false  predictions  issued  by  the  algorithm 
divided  by  the  total  number  of  predictions  issued. 

The  MOPR  for  microburst  prediction  are: 

1.  PFP  (i.e.,  actual  loss  less  than  15  knots)  of  <  0.10. 

2.  Prediction  lead  time  <  2  minutes  ±  2  minutes  prior  to  onset  of  microburst  for  60 
percent  of  predicted  events. 

A  test  suite  of  data  from  the  past  summer  consisting  of  20  days  from  Memphis  and  eight  days  from 
Orlando  was  used  to  score  the  algorithm.  The  prediction  algorithm  has  the  ability  to  run  in  two  modes,  unre¬ 
stricted  and  restricted.  In  unrestricted  mode,  the  algorithm  will  issue  every  prediction  that  is  made.  In  re¬ 
stricted  mode,  a  prediction  is  issued  only  if  it  overlaps  an  existing  wind  shear  alert  made  by  the  Microburst 
Detection  algorithm.  During  DemVal  the  algorithm  was  run  in  restricted  mode  in  Memphis  and  unrestricted 
mode  in  Orlando. 


TABLE  9  shows  the  total  statistics  for  each  site  in  both  unrestricted  and  restricted  modes.  The  POP 
comfortably  exceeds  the  MOPR  at  both  sites  in  restricted  and  unrestricted  modes.  The  MOPR  for  PFP  (0. 1 0 
at  15  knots)  is  met  in  the  restricted  mode  (0.0),  but  not  in  the  unrestricted  mode  (0.27).  The  MOPR  for  lead 
time  is  also  met.  The  detailed  analyses  are  presented  in  APPENDIX  B. 

The  majority  of  false  predictions  were  alarms  that  were  issued  too  late  and  actually  matched  micro¬ 
bursts  that  had  peaked  and  were  decreasing.  Work  is  continuing  to  fix  the  problem  of  late  predictions.  Other 
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false  alarms  were  found  on  days  when  the  algorithm  ran  for  an  extended  period  of  time  (i.e.,  overnight).  Ap¬ 
parently,  the  algorithm ’s  feedback  threshold  was  set  too  low.  When  new  weather  developed  the  following  day, 
false  alarms  resulted. 


TABLE  9. 

Performance  Statistics  for  the  Microburst  Prediction  Algorithm. 


SITE 

POP 

PFP 

AVERAGE 

LEAD  TIME  (SECS) 

RES 

UNRES 

RES* 

UNRES 

RES 

UNRES 

MEMPHIS 

0.80 

(20/25) 

0.80 

(20/25) 

0.00 

(0/155) 

0.33 

(132/404) 

126 

246 

ORLANDO 

0.56 

(14/25) 

0.64 

(16/25) 

0.00 

(0/118) 

0.19 

(63/329) 

58 

205 

TOTAL 

0.68 

(34/50) 

0.72 

(36/50) 

0.00 

(0/273) 

0.27 

(195/733) 

92 

226 

*  RES  is  restricted  mode;  UNRES  is  unrestricted  mode.  The  PFP  for  the  restricted  case  at  the  wind 
shear  level  is  0.0  because  a  prediction  must  overlap  a  wind  shear  event  to  be  issued. 
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6.  ITWS  GUST  FRONT  AND  WIND  SHIFT  PRODUCTS 


Like  the  current  operational  TDWR  gust  front  product,  the  ITWS  gust  front  and  wind  shift  products 
provide  air  traffic  controllers  and  supervisors  with  timely  reports  of  gust  front  location  and  strength,  and  pro¬ 
vides  planning  guidance  through  estimates  of  future  gust  front  positions  and  expected  wind  shifts.  In  addi¬ 
tion,  gust  fronts  may  contain  wind  shears  that  are  potentially  hazardous  to  landing  and  departing  aircraft. 
Thus,  the  product  also  enhances  safety  of  these  aircraft.  Unlike  the  TDWR  product,  the  ITWS  product  uses 
state-of-the-art,  knowledge-based  image  processing  techniques  to  detect  and  track  gust  fronts  in  Doppler 
radar  data.  This  technology  provides  more  gust  front  detections  (fewer  misses),  better  quality  detections 
(more  of  the  gust  front  length  is  detected),  and  more  consistent  detections  (better  estimate  capability)  than 
the  operational  TDWRs.  An  example  of  the  product  is  provided  in  Figure  13. 

6.1  GUST  FRONT  ALGORITHM 


A  detailed  description  of  the  gust  front  detection  algorithm  can  be  found  in  Troxel  and  Delanoy  ( 1 994). 
Briefly,  the  ITWS  approach  to  gust  front  detection  searches  radar  reflectivity  and  Doppler  velocity  data  for 
several  characteristic  signatures  in  the  data  that  may  indicate  the  presence  of  a  gust  front.  A  number  of  feature 
detectors  are  used;  one  detects  thin  lines  in  a  reflectivity  image,  another  “subtracts”  the  previous  reflectivity 
image  from  the  current  one  and  looks  for  motion,  and  another  feature  detector  looks  for  lines  of  convergence 
in  the  Doppler  velocity  map.  In  contrast,  the  TDWR  gust  front  detection  approach  uses  only  one  feature  detec¬ 
tor  (convergence)  to  find  gust  fronts. 

No  single  feature  detector  is  a  perfect  discriminator  in  all  situations;  not  all  signatures  are  always  pres¬ 
ent  and  other  phenomena  may  mimic  some  gust  front  features  (such  as  migrating  birds  and  thin  bands  of  light 
precipitation).  However,  the  combined  opinion  of  a  number  of  feature  detectors  provides  a  much  more  reli¬ 
able  confirmation  of  the  presence  of  a  gust  front.  The  ITWS  approach  combines  the  evidence  from  all  of  the 
feature  detectors.  After  the  evidence  has  been  combined,  a  threshold  is  applied  to  extract  the  gust  fronts. 

Once  the  gust  fronts  have  been  detected,  an  attempt  is  made  to  associate  each  gust  front  with  gust  fronts 
detected  in  earlier  images.  A  detection  history  is  maintained  to  allow  point-by-point  correspondence  and 
tracking  of  gust  fronts.  Propagation  speed  and  direction  are  calculated  for  each  associated  point  and  are  used 
to  extrapolate  the  position  of  that  point  into  the  future  to  produce  point-by-point  flexible  estimates  of  future 
gust  front  position.  For  each  gust  front  detection,  a  series  of  estimates  are  generated  for  1 -minute  intervals 
out  to  20  minutes.  For  each  detected  front,  the  Doppler  measurements  in  front  of,  within,  and  behind  the  front 
are  examined  to  estimate  the  wind  shear  hazard  and  the  wind  shift  associated  with  the  front.  LLWAS  sensor 
data  are  also  used  where  appropriate  to  improve  gust  front  wind  estimates  in  regions  where  Doppler  measure¬ 
ments  are  ambiguous  or  absent  (due  to  insufficient  return  signal). 


6.2  TECHNICAL  PERFORMANCE  ASSESSMENT  AND  RESULTS 

The  Gust  Front/Wind  Shift  products  must  meet  or  exceed  a  variety  of  operational  requirements.  Hence, 
there  are  a  number  of  performance  measures  that  are  needed  to  quantify  algorithm  performance. 


55 


57 


Numbered  arrows  indicate  expected  wind  shift  after  frontal  passage. 


Gust  Front  Location:  Two  different  metrics  are  used  to  assess  how  well  the  locations  of  gust  fronts 
are  reported  by  the  algorithm.  The  first  measure  is  a  crude  “hit/miss”  statistic  that  counts  a  detection  success¬ 
ful  if  any  part  of  the  detection  overlaps  a  2.7-nm  wide  gust  front  truth  region  identified  by  a  human  analyst. 
A  detection  is  counted  as  false  if  it  falls  completely  outside  of  any  truth  regions.  An  overall  POD  is  computed 
by  dividing  the  number  of  successfully  detected  fronts  by  the  number  of  fronts  identified  by  the  human  ana¬ 
lyst.  The  PEA  is  the  number  of  false  detections  divided  by  the  total  number  of  detections  (both  valid  and 
false). 


The  second  metric  better  quantifies  detection  quality  by  comparing  the  length  of  the  front  estimated 
by  the  algorithm  against  the  total  length  of  the  region  identified  by  the  human  analyst.  The  percent  of  length 
detected  (PLD)  is  the  length  detected  expressed  as  a  percent  of  the  length  delimited  by  the  human  analyst. 
The  percent  of  false  length  detected  (PFD)  reflects  the  fraction  of  total  detection  length  that  was  not  verified 
by  truth. 


Gust  Front  Location  Estimates:  Gust  front  location  estimates  are  scored  using  similar  metrics  as  for 
the  gust  front  location  described  above.  The  corresponding  hit/miss  metrics  are  PFId  (probability  of  generat¬ 
ing  a  correct  estimate,  given  a  detection)  and  PFIe  (probability  of  generating  a  correct  estimate,  given  an 
event),  while  the  corresponding  length-based  metrics  are  CFP  (Correct  Estimate  Probability)  and  FFP  (False 
Estimate  Probability).  The  scores  are  generated  by  comparing  10-  and  20-minute  estimate  positions  against 
the  2.7-nm  wide  gust  front  truth  regions  corresponding  to  the  valid  time  of  the  given  estimate. 


Scoring  procedures  for  gust  front  position  and  position  estimates  are  described  in  more  detail  in 
Klingle-Wilson  et  al,  1992. 


Wind  Shift:  The  wind  shift  product  is  currently  configured  to  estimate  the  wind  speed  and  direction 
that  will  occur  10  minutes  after  the  gust  front  crosses  the  airport  reference  point  (ARP).  To  score  how  well 
this  is  accomplished,  an  average  of  the  wind  shift  speed  and  direction  estimates  (generated  by  the  algorithm 
while  the  gust  front  was  between  10  and  20  minutes  away  from  the  airport)  is  compared  against  centerfield 
LLWAS  readings  taken  between  seven  and  13  minutes  after  gust  front  passage  over  the  ARP.  If  the  averaged 
wind  shift  speed  falls  within  the  error  bounds  described  in  TABLE  10,  the  wind  shift  estimate  is  considered 
a  hit  contributing  to  the  PAWS  (probability  of  accurate  wind  shift);  otherwise  it  is  a  miss,  contributing  to  the 
wind  shift  PEWS  (probability  of  erroneous  wind  shift).  For  example,  if  the  wind  shift  estimate  for  a  strong 
(greater  than  25  knots)  gust  front  is  within  ±  10.0  knots  and  ±  30  degrees  and  if  the  wind  shift  occurs  within 
7  to  13  minutes  after  the  gust  front  passes  the  ARP,  then  the  wind  shift  estimate  is  considered  a  hit. 

Wind  Shear  Estimate:  The  probability  of  generating  a  correct  wind  shear  alert  (PWS  A)  and  the  proba¬ 
bility  of  generating  a  wind  shear  alert  that  is  false  (PWSFA)  is  computed  by  comparing  algorithm  wind  shear 
hazard  estimates  against  analyst-generated  estimates  of  wind  shear  hazard  obtained  by  examination  of 
TDWR  Doppler  velocity  data,  from  pilot  reports  (if  available),  and/or  LLWAS  data  for  gust  fronts  that  im¬ 
pacted  airport  runways.  A  valid  alert  will  be  declared  if  the  reported  wind  shear  hazard  is  within  five  knots 
or  20  percent  of  the  actual  wind  shear  hazard,  whichever  is  greater.  A  false  alert  is  counted  if  the  actual  wind 
shear  hazard  is  less  than  15  knots,  and  the  reported  wind  shear  hazard  is  less  than  or  equal  to  20  knots.  If  the 
actual  wind  shear  hazard  is  less  than  or  equal  to  15  knots,  and  the  reported  wind  shear  hazard  exceeds  the 
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five  kiiot-20  percent  rule,  then  a  miss  is  declared.  Such  misses  will  reduce  the  PWS  A,  but  will  not  be  counted 
in  the  PWSFA. 


TABLE  10. 

Requirements  for  Wind  Shift  Estimate  as  a  Function  of  Wind  Shift  Strength. 


WIND  SHIFT 
STRENGTH 

PAWS 

PEWS 

WIND  SHIFT  ERROR 

ESTIMATED  TIME  OF 
ARRIVAL  ERROR 

>  25  knots 

70 

10 

±  1 0.0  knots,  ±  30  degrees 

±  3  minutes 

>  15  to  25  knots 

60 

10 

±  7.5  knots,  ±  30  degrees 

±  3  minutes 

5  to  15  knots 

50 

10 

±  5.0  knots,  ±  30  degrees 

- 

Given  the  algorithm  alert  classifications  described  above,  PWSA  is  computed  as  the  number  of  valid 
algorithm  wind  shear  alerts  divided  by  the  number  of  true  alerts.  The  PWSFA  is  defined  as  the  number  of 
false  wind  shear  alerts  divided  by  the  sum  of  valid  and  false  wind  shear  alerts.  The  gust  front  wind  shear  haz¬ 
ard,  or  headwind  gain  along  a  runway  corridor,  must  be  within  five  knots  or  20  percent  (whichever  is  greater) 
of  the  actual  gain. 


The  MOPR  for  the  gust  front  product  are: 


1 .  Predict  70  percent  of  gust  fronts  impacting  airport  with  wind  change  of  greater  than 
or  equal  to  15  knots  10  minutes  in  advance 

2.  Probability  of  false  10-minute  prediction  less  than  or  equal  to  0.10  for  gust  fronts 
with  wind  change  of  greater  than  or  equal  to  15  knots 


Gust  front  truth  data  were  generated  from  examination  of  TDWR  reflectivity  and  velocity  base  data 
plots  along  with  LLWAS  anemometer  time  series  plots  for  1 8  different  gust  front  events  which  occurred  dur¬ 
ing  a  1 0-hour  period  on  1 9  August  1 994  at  Orlando.  For  Memphis,  truth  data  were  generated  in  similar  fash¬ 
ion  for  30  gust  fronts  which  occurred  during  a  14-hour  period  on  8  July  1994.  The  truth  data  served  as  inputs 
to  an  automated  gust  front  scoring  utility  which  provided  performance  statistics  on  detection  accuracy  as  well 
as  10-minute  and  20-minute  estimate  accuracy.  Given  the  limited  number  of  different  days  analyzed  so  far, 
detection  performance  results  presented  here  should  be  considered  preliminary. 
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TABLE  1 1  summarizes  gust  front  detection  performance  for  the  ITWS  gust  front  detection  algorithm 
for  the  Memphis  and  Orlando  cases.  As  is  typical  for  Orlando,  there  were  no  strong  or  severe  gust  fronts  on 
this  particular  day.  Detection  performance  was  excellent,  both  in  terms  of  the  event-based  metrics  and  in 
terms  of  total  length  detected.  Although  several  gust  fronts  with  strong  and  severe  strengths  were  observed 
during  the  summer,  the  gust  fronts  analyzed  for  the  July  08  case  were  all  weak  or  moderate  strength.  Detection 
probabilities  fell  slightly  for  the  Memphis  case,  which  had  an  overall  POD  of  0.89.  In  Memphis,  there  were 
more  gust  fronts  embedded  within  storm  cells.  With  embedded  fronts,  the  only  evidence  is  the  velocity  con¬ 
vergence  signature  (no  distinct  reflectivity  thin  line),  and  as  a  result,  detections  of  some  of  the  weaker  events 
were  sometimes  delayed  as  the  algorithm  required  additional  “looks”  at  the  event  to  build  sufficient  confi¬ 
dence  before  reporting  the  gust  front. 


TABLE  11. 

Gust  Front  Detection  Performance  for  the  ITWS  Gust  Front  Detection  Algorithm 


1  Strength 

POD 

PFA 

PLD 

PFD  1 

1  Memphis  | 

Weak 

0.88 

0.81 

- 

Moderate 

1.00 

- 

0.98 

- 

Strong 

- 

- 

- 

- 

Severe 

- 

- 

- 

All  Memphis 

0.89 

0.06 

0.82 

0.24 

1  Orlando  | 

Weak 

0.94 

- 

0.85 

Moderate 

1.00 

- 

0.85 

- 

Strong 

- 

- 

- 

- 

Severe 

- 

- 

- 

- 

All  Orlando 

0.95 

0.05 

0.85 

0.12 

False  alarms  at  both  locations  tended  to  be  sporadic  and  short-lived.  Most  were  dropped  after  a  single 
algorithm  processing  interval  (five  minutes).  Primary  causes  of  false  detections  were  noisy  data,  alerting  on 
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radar  features  that  were  weak  (e.g.  weak  convergence,  limited  motion,  length  less  than  0.5  nm),  and  inap¬ 
propriate  “coasting”  of  detections  after  the  event  had  dissipated. 


TABLE  12  lists  PFId  results  for  Memphis  and  Orlando.  The  results  show  that  in  general,  position  esti¬ 
mates  were  reliably  generated  once  a  detection  had  been  issued.  More  than  80  percent  of  the  10-minute  posi¬ 
tion  estimates  for  all  gust  front  detected  in  Memphis  and  Orlando  were  validated.  As  expected,  performance 
for  20-minute  estimates  decreased,  especially  at  Memphis,  due  to  the  increased  likelihood  of  changes  in  gust 
front  propagation  or  strength  over  the  longer  estimation  period  (the  Memphis  gust  fronts  exhibited  stronger 
variability  over  time  due  to  more  vigorous  storm  dynamics,  making  estimates  more  challenging  for  that  site). 


TABLE  12. 

PFId  Results  for  the  ITWS  Gust  Front  Detection  Algorithm. 


Weak 

Moderate 

Strong 

Severe 

All 

False 

Memphis  (lO-min) 

0.80 

1.00 

- 

- 

0.81 

0.15 

Memphis  (20-min) 

0.68 

0.91 

- 

- 

0.69 

0.28 

Orlando  (10-min) 

0.88 

1.00 

- 

- 

0.89 

0.07 

Orlando  (20-min) 

0.81 

0.57 

- 

- 

0.80 

0.22 

Overall  false  estimate  probabilities  were  respectable,  but  are  slightly  higher  than  desired  for  the  Mem¬ 
phis  10-minute  estimates.  Additional  investigation  into  the  causes  of  false  estimates  (at  both  sites)  revealed 
that  a  substantial  fraction  of  the  false  estimates  were  associated  with  weaker  fronts  that  are  often  short-lived 
and  may  dissipate  even  before  the  10-minute  estimation  period  has  elapsed.  Moreover,  the  MOPR  for  gust 
front  position  estimates  calls  for  correct  10-minute  advance  predictions  of  70  percent  or  more  of  gust  fronts 
impacting  the  airport  with  a  wind  change  equaling  or  exceeding  15  knots,  with  a  probability  of  false  predic¬ 
tion  for  the  same  fronts  not  to  exceed  0. 1 . 


To  better  determine  how  well  the  MOPR  was  met  for  the  position  estimates,  a  second  evaluation  was 
conducted  with  weaker  gust  front  detections  (wind  shift  strength  less  than  15  knots)  excluded  from  the  data 
set.  However,  since  most  Orlando  gust  fronts  tend  to  be  weaker  than  the  15  knot  threshold,  this  resulted  in 
only  1 6  individual  detections  remaining  from  the  1994  data  set  for  evaluation.  Because  of  the  limited  number 
of  non-weak  Orlando  gust  fronts  evaluated,  results  for  the  Orlando  1994  tests  are  inconclusive.  Since  data 
were  available  from  similar  real-time  tests  conducted  in  1 993  in  Orlando,  combined  results  for  the  1 993  and 
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1994  operations  are  shown  in  TABLE  13.  This  combined  data  set  provided  over  300  individual  detections 
with  which  to  perform  the  evaluation.  The  10-minute  estimates  of  gust  fronts  with  significant  shear  at  Mem¬ 
phis  and/or  Orlando  met  the  MOPR. 


TABLE  13. 

PFld  Results  for  the  ITWS  Gust  Front  Detection  Algorithm. 
(Wind  Shift  Strength  >  15  knots) 


1  Location 

Correct 

False  I 

1  10-minute  Estimates  | 

Memphis 

0.95 

0.05 

Orlando  (93  and  94) 

0.83 

0.09 

I  20-minute  Estimates  | 

Memphis 

0.90 

0.12 

Orlando  (93  and  94) 

0.72 

0.22 

Position  estimate  performance  for  all  gust  front  strengths  was  also  assessed  independent  of  detection 
performance.  PFIe  are  shown  for  Memphis  and  Orlando  gust  fronts  in  TABLE  14.  In  Memphis,  accurate 
1 0-minute  estimates  were  generated  for  73  percent  of  the  gust  fronts  that  occurred,  while  83  percent  of  the 
Orlando  gust  fronts  were  accompanied  by  accurate  10-minute  estimates.  Compared  to  10-minute  estimates, 
PFIe  values  for  the  20-minute  estimates  fell  by  less  than  10  percent  at  each  of  the  two  sites.  As  expected,  the 
number  of  false  20-minute  estimates  was  higher  in  Memphis  than  in  Orlando  due,  once  again,  to  the  more 
vigorous  storm  dynamics  at  Memphis. 


TABLE  15  presents  corresponding  length-based  statistics  (CFP,  FFP)  for  10-minute  and  20-minute 
estimates.  The  length-based  statistics  echo  the  trends  seen  and  discussed  for  the  event-based  statistics  in 
TABLE  14. 
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TABLE  14. 

PFIe  Results  for  lO-minute  and  20-minute  Estimates. 


Weak 

Moderate 

Strong 

Severe 

All 

False 

Memphis  (10-min) 

0.70 

0.92 

- 

- 

0.73 

0.17 

Memphis  (20-min) 

0.62 

0.76 

- 

- 

0.64 

0.30 

Orlando  (10-min) 

0.82 

1.00 

- 

- 

0.83 

0.11 

Orlando  (20-min) 

0.76 

0.44 

- 

- 

0.74 

0.17 

TABLE  15. 

CFP  Results  for  10-minute  and  20-minute  Estimates. 


Weak 

Moderate 

Strong 

Severe 

All 

FFP 

Memphis  (10-min) 

0.53 

0.93 

- 

0.56 

0.43 

Memphis  (20-min) 

0.37 

0.76 

- 

- 

0.40 

0.58 

Orlando  (10-min) 

0.73 

0.72 

- 

- 

0.73 

0.21 

Orlando  (20-min) 

0.62 

0.42 

- 

- 

0.61 

0.32 

Wind  shift  estimates  and  wind  shear  hazard  reports  were  compared  against  LLWAS  centerfield 
anemometer  data  for  18  gust  fronts  that  were  tracked  across  the  Memphis  airport  during  June  and  July  of 
1 994.  Performance  statistics  were  computed  using  the  performance  measures  described  previously.  Truth  and 
algorithm  data  values  were  rounded  to  the  nearest  five  knots  for  wind  shift  speed  and  wind  shear  reports,  and 
to  the  nearest  1 0  degrees  for  the  wind  shift  direction.  Results  presented  here  should  be  considered  preliminary 
given  the  limited  number  of  events  examined  to  date.  APPENDIX  C  lists  the  wind  shift  and  wind  shear  esti¬ 
mates  used  in  this  analysis. 


Wind  shear  hazard  estimates  produced  by  MIGFA  met  requirements  for  all  18  of  the  gust  fronts  ex¬ 
amined,  resulting  in  aPWSA  of  1.0  (18/18)  and  PWSFA  of  0.0  (0/18).  Performance  results  for  the  wind  shift 
estimate  as  a  function  of  observed  wind  shift  strength  are  presented  in  TABLE  1 6.  Note  that  for  the  wind  shift 
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PEWS,  a  “false  alarm”  does  not  mean  that  a  wind  shift  estimate  should  not  have  been  made,  only  that  its 
reported  value  was  declared  to  be  inaccurate. 


TABLE  16. 

Gust  Front  Wind  Shift  Accuracy  (Direction  Error  Tolerance  Equals  30  Degrees). 


STRENGTH  CATEGORY 

PAWS 

PEWS 

<  5  knots 

1.00(1/1) 

0.00  (0/1) 

>  5  to  15  knots 

0.60  (9/15) 

0.40  (6/15) 

>  15  to  25  knots 

1.00(1/1) 

0.00  (0/1) 

>  25  knots 

1.00  (1/1) 

0.00  (0/1) 

ALL 

0.67  (12/18) 

0.33  (6/18) 

With  the  exception  of  the  five  to  15  knot  category,  MIGFA  appears  to  meet  the  desired  operational  re¬ 
quirements,  albeit  with  only  one  gust  front  comparison  in  each  of  the  other  categories.  MIGFA  appears  to 
fall  short  of  the  requirements  for  PEWS  in  the  five  to  1 5  knot  category,  which  also  contained  the  largest  num¬ 
ber  of  fronts  (15).  Circumstances  surrounding  the  six  inaccurate  wind  shift  estimates  are  summarized  in 
TABLE  17. 


Our  limited  data  set  included  a  number  of  complex  scenarios  including  gust  fronts  accompanied  by 
high  frequency  wind  oscillations  from  gravity  wave  phenomena,  and  gust  fronts  exhibiting  rapid  develop¬ 
ment  and  decay.  In  addition,  there  are  at  least  two  cases  where  the  TDWR  and  LLWAS  reported  large  differ¬ 
ences  in  observed  (truth)  wind  directions,  so  there  is  disagreement  even  between  sensors  observing  the  same 
phenomenon  (MIGFA  relies  mostly  on  TDWR  data  for  wind  shift  estimates). 


Another  complicating  factor  in  the  wind  shift  analysis  is  the  effect  of  timing  (ETA)  errors  on  the  accura¬ 
cy  of  the  wind  shift  estimate.  Most  gust  front  wind  shifts  occur  over  a  transitional  period  of  up  to  several 
minutes,  with  winds  gradually  veering  and  strengthening.  In  addition,  some  gust  fronts  do  not  generate  per¬ 
sistent  wind  shifts.  Over  the  time  period  of  interest,  the  winds  behind  the  front  may  not  settle  on  a  particular 
wind  speed  or  direction,  making  estimates  difficult  to  verify  as  it  is  not  clear  exactly  when  the  ’’shift”  has  been 
completed.  As  a  result,  comparisons  of  ’’truth”  against  algorithm  estimates  can  be  significantly  altered  by 
relatively  small  changes  in  location  and  times  over  which  the  truth  and  algorithm  report  samples  are  selected. 
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In  the  future,  a  more  thorough  examination  of  the  issue  of  timing  errors  and  their  effects  on  wind  shift  estimate 
is  plaimed. 


TABLE  17. 

Summary  of  MIGFA  Wind  Shift  Estimate  Probiems 


Date 

Time 

(GMT) 

Comment 

06/07/94 

0523 

Poor  radar  viewing  angle  led  to  poor  direction  estimate. 

06/16/94 

2123 

Bad  direction  estimate  (good  speed  estimate). 

06/28/94 

1420 

Gust  front  weakened  rapidly  after  estimate. 

07/01/94 

0244 

Variable  winds  due  to  gravity  wave  phenomenon. 

07/05/94 

1959 

LLWAS  direction  differed  by  130  degrees  (LLWAS  disagrees  with  TDWR  ob¬ 
servation  by  140  degrees).  Also,  wind  speed  =  5  knots,  so  winds  were  light 
and  variable  even  after  gust  front  passage. 

07/08/94 

2148 

LLWAS  direction  differed  by  40  degrees.  However,  good  agreement  with 
TDWR  (only  10  degrees  difference). 

Overall  performance  of  the  gust  front/wind  shift  algorithm  appears  to  be  very  good.  MOPR  were  met 
or  exceeded.  The  higher-than-acceptable  false  alarm  rate  for  the  wind  shift  estimates  in  the  five  to  15  knot 
strength  category  does  not  meet  the  stated  goal.  The  wind  direction  component  of  the  wind  shift  vector  was 
found  to  be  the  most  significant  contributor  to  the  errors  and  and  these  errors  were  often  associated  with 
events  that  were  complex  in  nature.  The  present  data  set  for  this  part  of  the  evaluation  is  quite  limited  (Mem¬ 
phis  only)  and  contains  a  number  of  complex  cases.  Additional  cases,  including  some  from  Orlando,  should 
be  examined  to  more  thoroughly  assess  wind  shift  estimate  performance. 


Based  on  the  results  presented  here,  as  well  as  anecdotal  evidence  gathered  during  the  course  of 
operations  last  summer,  Memphis  proved  to  be  a  more  challenging  environment  for  gust  front  detection  than 
Orlando.  This  was  expected  since  the  majority  of  Orlando  gust  fronts  originate  from  isolated  air  mass 
thunderstorms,  and  they  usually  propagate  into  clear  air  away  from  the  generating  storm  soon  after  formation. 
As  a  result,  the  Orlando  fronts  are  often  accompanied  by  reflectivity  thin  lines  as  well  as  velocity 
convergence,  making  detection,  tracking,  and  prediction  easier.  By  contrast,  several  of  the  Memphis  gust 
front  events  were  associated  with  more  vigorous  weather,  including  squall  lines  and  synoptic-scale  frontal 
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passages.  Many  of  the  Memphis  gust  fronts  were  embedded  in  precipitation  and  the  associated  gust  front 
characteristics  (strength,  propagation  velocity,  longevity)  were  less  stable.  MIGFA’s  gust  front  position 
estimation  capability  was  found  to  be  impacted  by  these  phenomenological  instabilities,  especially  the 
20-minute  estimate  capability. 


67 


7.  ITWS  TERMINAL  WINDS  PRODUCT 


The  rrWS  Terminal  Winds  product  provides  frequently  updated  (every  five  minutes)  estimates  of  the 
horizontal  wind  at  various  altitudes  for  points  of  interest  to  ATC  users  (for  example,  at  the  arrival  and  depar¬ 
ture  gates  and  at  tum-on  to  final  approach).  During  the  1994  demonstration,  the  terminal  winds  product  may 
be  used  by  ATC  users  for  traffic  management  and  situational  awareness.  For  the  initial  ITWS  deployment, 
the  Terminal  Winds  information  will  be  used  also  by  the  Terminal  Air  Tralfic  Control  Automation  system 
and  the  ITWS  gust  front  detection.  Future  applications  for  this  information  include  the  ITWS  Ceiling  &  Visi¬ 
bility  prediction.  Runway  Winds  prediction.  Thunderstorm  Growth  and  Decay,  and  Wake  Vortex  Advisory 
systems.  An  example  of  the  Terminal  Winds  product  is  provided  in  Figure  14. 


7. 1  TERMINAL  WINDS  ESTIMATION  ALGORITHM 


Currently,  wind  estimates  are  produced  for  two  grids.  The  first  grid  has  a  horizontal  resolution  of  about 
5  nm  and  a  vertical  resolution  of  about  1500  feet  near  the  surface,  increasing  with  altitude  to  about  3,700  feet 
at  30,000  feet.  This  product  is  generated  for  a  135  nm  x  135  nm  region  centered  on  the  ARP,  extending  verti¬ 
cally  to  about  53,000  feet  and  is  updated  every  30  minutes.  Figure  15  shows  the  analysis  regions  and  sensor 
locations  for  the  Orlando  International  Airport  in  1992,  when  a  slightly  smaller  5  nm  grid  was  used.  The  pri¬ 
mary  sources  of  data  for  the  furst  grid  are  the  Meteorological  Data  Collection  and  Recording  System 
(MDCRS)  reports  from  commercial  aircraft  arriving  and  departing  the  airport  and  the  Aviation  Gridded  Fore¬ 
cast  System  (AGFS),  a  weather  forecast  model.  (The  AGFS  is  produced  operationally  by  the  NWS  National 
Meteorological  Center  as  the  Rapid  Update  Cycle  product.  For  the  DemVal,  the  prototype  AGFS  produced 
by  the  National  Oceanic  and  Aeronautic  Administration  Forecast  Systems  Laboratory  was  used. 


The  second  grid  is  nested  in  the  first.  It  has  a  horizontal  resolution  of  1  nm  and  the  same  vertical  resolu¬ 
tion.  The  second  grid  covers  a  65  run  x  65  run  region  centered  on  the  ARP,  extends  vertically  to  about  18,000 
ft  and  is  updated  every  five  minutes.  The  primary  sources  of  data  for  the  1  nm  product  are  the  Doppler  data 
provided  by  the  TD\^^  and  NEXRAD  weather  radars  and  the  5  nm  resolution  analysis. 


The  full  set  of  data  sources  used  to  compute  wind  estimates  for  the  Terminal  Winds  product  are: 


1 .  AGFS  -  a  national  scale  weather  forecast  model  with  a  32  nm  horizontal  resolution  and 
3  hr  update  rate. 

2.  TDWR  Doppler  radar  -  single  (radial)  velocity  component  measurements  with  a  480  ft 
horizontal  resolution  and  a  five  minute  update  rate. 

3 .  NEXRAD  Doppler  radar  -  single  (radial)  velocity  component  measurements  with  a  800 
ft  horizontal  resolution  and  a  six  minute  update  rate. 
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Figure  15. 1992  Orlando  Terminal  Winds  domains  and  sensor  locations. 


4.  MDCRS  Aircraft  measurements  -  vector  measurements  of  the  horizontal  wind  taken  at 
random  points  and  times  along  flight  paths,  with  a  data  latency  of  at  least  15  minutes. 

5 .  LLWAS  -  six  to  20  vector  measurements  at  the  airport  surface  with  a  1  to  1 .5  nm  horizon¬ 
tal  resolution  and  a  10  second  update  rate. 

6.  Automated  Surface  Observing  Stations  -  widely  spaced  vector  measurements  at  the  sur¬ 
face  with  rapid  update  rate. 

The  Doppler  radars  provide  accurate  and  dense  measurements  in  regions  with  enough  reflectors  (for 
example,  bugs  and  dust)  for  the  radars  to  gather  returns.  This  limits  the  vertical  extent  of  dense  Doppler  data 
on  most  days  to  7,000  ft  or  less  in  the  summer.  In  the  dry  clear  air  of  the  winter,  the  vertical  extent  of  good 
Doppler  return  will  be  even  lower.  Doppler  radars  are  also  limited  to  measuring  only  the  component  of  the 
wind  either  towards  or  away  from  the  radar.  That  is,  if  the  radar  is  looking  due  north  and  the  wind  is  from 
the  west,  the  radar  will  measure  the  north  component  of  the  wind,  which  is  zero.  The  radar  correctly  measures 
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the  north  component  but  does  not  provide  information,  at  that  point,  on  the  west  component.  The  west  compo¬ 
nent  must  be  derived  from  other  data  sources,  including  other  Doppler  radars,  anemometers,  and/or  MDCRS. 

The  ITWS  Terminal  Winds  receives  data  from  a  numbers  of  sources  which  provide  information  of  dif¬ 
fering  content,  update  rate,  and  quality.  The  analysis  must  properly  assemble  this  data  to  provide  accurate 
estimates  of  the  wind  at  each  point  of  the  analysis  grid.  This  is  done  in  a  two  step  cascade-of-scales  process. 
At  each  step,  finer  resolution  wind  estimates  are  computed  using  a  least  squares  statistical  technique  to  mini¬ 
mize  the  errors  in  the  final  wind  estimates.  Figure  16  shows  the  data  flow  for  this  process. 

The  first  step  in  the  cascade-of-scales  is  to  compute  the  5  nm  resolution  product.  The  AGFS  wind  fore¬ 
casts  are  interpolated  to  the  5  nm  grid  and  then  refined  using  all  of  the  available  data.  This  is  done  every  30 
minutes. 


AGFS  FORECAST  MODEL  — 

TDWR  - 1 

NEXRAD  - 

AIRCRAFTrMDCARS - 

LLWAS  - , 

AUTO  SURFACE  OBS  - 

HOURLY  SURFACE  OBS  - 


135  NMX 135  NM  GRID 
5  NM  GRID  RESOLUTION 
19  LEVELS  (53,000  FT) 

30  MINUTE  UPDATE 


5  NM  INTERPOLATION 


5NM 

RESOLUTION 
WIND  FIELD 


TDWR 


NEXRAD 


LLWAS 


AUTO  SURFACE  OBS 


1  NM  INTERPOLATION 


65  NM  X  65  NM  GRID 

I  NM  GRID  RESOLUTION 

II  LEVELS  (18,000  FT) 
FIVE  MINUTE  UPDATE 


1  NM 

RESOLUTION 
WIND  FIELD 


Figure  16.  Dataflow  for  Terminal  Winds. 
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The  second  step  is  to  compute  the  1  nm  resolution  product.  In  this  step,  the  current  5  nm  product  is 
interpolated  to  the  1  nm  grid  and  then  refined  using  only  the  rapid  update  data  (that  is,  Doppler  data  and  auto¬ 
mated  surface  observations)  every  five  minutes. 


7.2  TECHNICAL  PERFORMANCE  ASSESSMENT  AND  RESULTS 


The  Terminal  Winds  analysis  produces  a  three  dimensional  grid  of  horizontal  wind  vectors  for  a  region 
covering  the  terminal  area.  The  accuracy  of  these  wind  vectors  is  evaluated  by  comparing  them  with  observa¬ 
tions  of  the  wind  in  the  terminal  area.  These  observations  are  obtained  from  two  sources.  One  source  is  a 
dual-Doppler  analysis  of  TDWR  and  NEXRAD  data,  and  other  is  MDCRS.  Each  observation  is  matched 
with  the  analysis  vector  having  the  same  time  and  grid  coordinates.  These  matched  pairs  of  vectors  are  com¬ 
piled  into  two  data  bases;  one  for  dual-Doppler  truth  and  one  for  MDCRS  truth.  These  databases  are  analyzed 
using  the  SAS  data  analysis  system.  The  products  of  this  analysis  are  a  set  of  statistics  and  histograms  charac¬ 
terizing  the  differences  between  the  observations  and  analysis  vectors.  The  standard  evaluation  consists  of 
statistics  on  the  norm  of  the  vector  difference  between  the  observations  and  analysis,  and  histograms  showing 
the  the  distributions  of  norm  or  the  vector  difference,  U  component  difference,  V  component  difference, 
speed  difference,  and  direction  difference. 

The  MOPR  for  the  Terminal  Winds  product  are: 

1 .  horizontal  resolution:  5  nm  within  30  nm  beyond  the  TRACON  boundary  and  below 
23,000  ft 

2.  vertical  resolution:  50  millibars 

3.  ±10  knots  80%  of  time  in  regions  and  at  times  when  both  TDWR  and  NEXRAD  have 
valid  velocity  data 

During  DemVal,  the  1-nm  resolution  product  was  created.  This  product  exceeds  the  MOPR  for  hori¬ 
zontal  and  meets  the  MOPR  for  vertical  resolution. 

The  results  of  the  evaluation  of  the  terminal  winds  algorithm  are  provided  in  TABLE  18.  The  table 
contains  the  statistics  on  the  norm  of  the  vector  difference  between  dual-Doppler  and  each  algorithm.  The 
columns  Q75  and  Q25  represent  the  75th  and  25th  quartiles,  respectively.  There  were  so  few  MDCRS  data 
in  Memphis  that  they  can  not  support  a  meaningful  estimate  of  algorithm  performance. 


The  results  of  the  comparison  to  dual-Doppler  over  10  days  show  that  the  Terminal  Winds  product  1) 
has  vector  errors  less  than  2  knots  75  percent  of  the  time  in  regions  that  have  data  from  both  TDWR  and  NEX¬ 
RAD,  and  2)  does  not  degrade  the  forecasts  from  AGES.  The  number  of  available  MDCRS  reports  over  the 
1 0  days  was  only  64.  The  comparison  of  wind  fields  to  MDCRS  shows  a  greater  agreement  to  Terminal  Winds 
than  AGES  forecasts,  but  the  aircraft  reports  were  too  few  in  number  to  accurately  quantify  performance. 
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TABLE  18. 

Performance  Statistics  for  the  Terminal  Winds  Algorithm. 


Analysis 

Grid 

Mean 

Variance 

Q80 

Median 

Q25 

RMS-Error 

AGFS 

CO 

4.2 

4.7 

3.1 

2.0 

3.9 

5  NM 

1.4 

1.9 

2.0 

1.0 

0.6 

2.0 

1  NM 

0.6 

0.4 

1.0 

0.5 

0.3 

0.9 
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8.  ITWS  AIRPORT  LIGHTNING  PRODUCT 


The  ITWS  Airport  Lightning  product  provides  an  indication  that  lightning  has  been  detected  near  a 
user-specified  location.  An  example  of  the  product  is  provided  in  Figure  17.  The  presence  of  lightning  is 
indicated  by  the  alert  panel  in  the  upper  right  comer  of  the  display.  This  product  may  be  used  by  ATC  users 
to  determine  when  to  switch  to  back-up  generator  power  and  by  airline  users  to  help  in  making  decisions  with 
respect  to  refueling  and  other  ground  operations. 


8.1  LIGHTNING  WARNING  PANEL  ALGORITHM 


The  National  Lightning  Detection  Network  (NLDN)  provides  the  raw  data  that  are  used  by  the  light¬ 
ning  processor.  These  raw  data  reports  give  the  time,  location  and  strength  of  lightning  strikes  (primarily 
cloud-to-ground  lightning).  These  data  are  quality-edited  by  comparing  the  reported  location  of  the  light¬ 
ning  strike  with  an  ASR-9  weather  channel  image  collected  at  the  same  time.  Reported  locations  are  accepted 
or  rejected  depending  on  their  proximity  to  any  weather  observed  on  the  radar  image. 


The  lightning  warning  panel  in  the  upper  right  comer  of  the  Situation  Display  is  illuminated  whenever 
lightning  is  detected  within  a  user-specified  distance  of  a  user-specified  reference  location.  Generally,  the 
reference  location  is  taken  to  be  the  ARP  and  the  distance  is  either  5  nm  for  ATC  users  or  20  nm  for  airline 
companies.  The  specification  of  a  reference  location  and  distance  defines  a  circular  region  called  the  Critical 
Region.  The  warning  panel  is  illuminated  when  any  edited  lightning  strike  report  is  detected  within  this  Criti¬ 
cal  Region  and  it  will  remain  illuminated  for  five  minutes  after  the  last  detection  in  the  Critical  Region. 


8.2  TECHNICAL  PERFORMANCE  ASSESSMENT  AND  RESULTS 


The  measures  of  performance  are  product  availability  (the  fraction  of  the  time  that  the  lightning  data 
is  available),  latency  of  the  lightning  warning  panel  (amount  of  time  between  the  receipt  of  a  lightning  event 
and  its  transmission  to  the  warning  panel),  and  meteorological  consistency  (the  degree  of  association  of  light¬ 
ning  and  precipitation). 


Product  Availability:  The  number  of  status  messages  received  per  hour  and  the  status  messages  them¬ 
selves  indicate  the  amount  of  data  loss.  System  logs  reveal  that  the  lighming  data  were  available  to  the 
end-user  99.2  percent  of  the  time. 


Latency  of  the  Lightning  Warning  Panel:  More  than  99  percent  of  all  lightning  reports  arrived  at 
the  input  to  the  ITWS  lightning  processor  within  15  seconds  of  occurrence.  These  data  were  then  collected 
into  25  second  intervals  and  sent  to  the  display.  Therefore,  the  maximum  latency  of  the  warning  panel  was 
about  40  seconds. 
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Figure  1 7.  Example  of  the  Lightning  Detection  product.  The  yellow  box  in  the  upper  right  corner  is  illuminated  when  cloud-to-ground  lightning  is  detected  within 
user-specified  distance  (20  nmfor  ATC  users;  5  nm  for  other  users)  of  the  airport  reference  point. 


Accuracy  of  NLDN:  Two  studies  of  the  performance  of  the  NLDN  were  performed;  one  in  1992  in 
Orlando  and  another  in  1 994  in  Albuquerque,  NM.  The  locational  accuracy  of  the  NLDN  was  ascertained 
by  comparing  the  reported  locations  of  lightning  strikes  with  the  locations  provided  by  the  three-dimension¬ 
al,  total  lightning  mapping  system  operated  by  the  French  Office  Nationale  D’Etudes  et  de  Recherches  Aero- 
spatiales  (ONERA).  These  comparisons  indicate  that  the  root-mean-square  separation  between  the  locations 
was  4  nm  in  Orlando  and  4.6  nm  in  Albuquerque. 


The  overlays  of  lightning  and  ASR-9  weather  reveal  the  percentage  of  lightning  reports  that  can  be 
considered  erroneously  decoded.  A  false  alarm  was  declared  if  the  reported  position  of  the  lightning  was  more 
than  1 1  nm  from  any  ASR-9  observed  weather  level  1  or  greater.  Less  than  0. 1  percent  of  the  lightning  reports 
were  rejected  as  false  alarms  during  the  Memphis  DemVal. 


NLDN  saw  70  percent  of  all  cloud-to-ground  strokes  detected  by  the  ONERA-3D  total  lightning 
mapper.  The  detection  efficient  rose  to  80  percent  when  considering  just  the  strongest  50  percent  of  the  cloud- 
to-ground  strokes  detected. 


9.  TERMINAL  WEATHER  TEXT  MESSAGE  PRODUCT 


The  Terminal  Weather  Text  Message  product  shows  ATC  users  the  messages  that  are  being  delivered 
directly  to  aircraft  via  ACARS  data  link.  An  important  goal  of  the  ITWS  program  is  to  improve  pilot  situa¬ 
tional  awareness  while  decreasing  the  controller  workload  involved  in  providing  weather  information  to  pi¬ 
lots.  As  part  of  this  effort,  text-based  Terminal  Weather  messages  will  be  provided  directly  to  aircraft  via  the 
ACARS  data  link.  These  Terminal  Weather  messages  provide  a  summary  of  the  weather  conditions  around 
the  airport  based  on  the  ITWS  products  displayed  on  the  Situation  Display.  This  product  allows  ATC  users 
to  monitor  the  ITWS  Terminal  Weather  messages  being  delivered  to  aircraft  for  shared  situational  awareness. 
An  example  of  the  product  is  provided  in  Figure  18. 


9.1  PRODUCT  GENERATION 


The  Terminal  Weather  Text  Message  Product  is  generated  from  ITWS  products  used  for  display  on  the 
Situation  Display.  The  Storm  Cell  Information  product  is  used  to  locate  storms  within  the  terminal  area.  The 
Storm  Motion  product  is  used  to  report  the  general  motion  of  all  storms  within  the  terminal  area.  The  runway 
impact  product  is  used  to  generate  the  microburst  and  gust  front  information.  Finally,  the  ITWS  Precipitation 
product  is  used  to  generate  the  precipitation  impacts  at  the  airport. 


The  text  of  the  Text  Message  product  is  divided  into  three  sections;  Airport  Impacts,  Terminal  Weather, 
and  Expected  Airport  Weather.  The  Airport  Impacts  section  is  indicated  by  an  asterisk  (*)  and  lists  the  worst 
weather  affecting  any  of  the  operational  runways.  For  example: 

^MODERATE  PRECIP 
BEGAN  2054 

If  a  microburst  is  impacting  one  runway  while  heavy  precipitation  is  impacting  another  runway,  the  text  mes¬ 
sage  product  will  issue  information  about  the  microburst  (the  most  hazardous  weather  impacting  the  airport). 
The  message  would  look  like: 

*MICROBURST  ALERTS 
30  KT  LOSS 
BEGAN  2105 


The  Terminal  Weather  section  is  indicated  by  a  dash  (-)  and  identifies  the  three  closest  storms  to  the 
airport.  For  each  storm,  the  text  message  lists  the  minimum  distance  in  nautical  miles  from  the  ARP  to  the 
weather,  the  direction  of  the  storm  from  ARP,  the  storm  intensity  based  on  the  highest  weather  level  in  the 
storm  (MOD  for  level  2,  HVY  for  level  3  and  greater),  and  HAIL  (if  present).  The  last  line  in  this  section 
states  the  average  motion  of  all  storms  in  the  TRACON.  An  example  of  the  Terminal  Weather  message  is: 
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delivered  directly 


-STORM(S) 

APRT  MOD  PRECIP 
INM  W  HVY  PRECIP 
17NM  N  MOD  PRECIP 
MOVG  E  AT  9  KT 


(moderate  precipitation  at  airport) 

(heavy  precipitation  1  nm  west  of  ARP) 
(moderate  precipitation  17  nm  north  of  ARP) 
(moving  east  at  nine  knots) 


The  Expected  Airport  Weather  section  is  indicated  by  a  dot  (.)  and  reports  any  expected  precipitation 
that  will  impact  the  operational  runways.  This  section  uses  the  ITWS  Precipitation  product  and  the  average 
motion  of  all  storms  to  estimate  the  time  at  which  the  precipitation  will  begin  on  a  runway.  The  message  will 
list  the  worst  weather  expected  on  the  runways.  For  example,  if  moderate  precipitation  is  expected  on  one 
runway  and  heavy  precipitation  is  expected  on  another  runway,  the  message  will  provide  the  information  rela¬ 
tive  to  the  heavy  precipitation.  This  section  does  not  currently  attempt  to  predict  wind  shear  or  microburst 
activity.  An  example  of  a  message  in  the  Expected  Airport  Weather  section  is: 

.EXPECTED  MOD  PRECIP 
BEGIN  2055 


If  no  weather  is  found  within  20  nm  of  the  ARP,  the  text  message  is: 
.NO  STORMS  WITHIN  20NM 


9.2  TECHNICAL  PERFORMANCE  ASSESSMENT  AND  RESULTS 


The  precipitation  impact  messages  are  used  as  truth.  An  event  is  defined  as  level  2  or  greater  precipita¬ 
tion  impacting  at  least  one  arena  for  at  least  five  minutes.  An  event  is  correctly  predicted  if  the  event  is  preced¬ 
ed  by  at  least  one  prediction  in  the  1 5  minutes  prior  to  the  onset  of  the  event.  The  POP  is  defined  as  the  number 
of  correct  predictions  divided  by  the  number  of  events.  A  false  alarm  is  a  prediction  of  an  event  that  does  not 
verify  (i.e.,  precipitation  does  not  impact  an  arena).  The  PEP  is  the  number  of  false  alarms  divided  by  the 
total  number  of  predictions. 

For  each  event,  it  is  verified  automatically  that  an  expected  precipitation  message  {i.e.,  a  prediction) 
occurred  at  least  15  minutes  prior  to  the  onset  of  the  event.  For  each  event,  the  predicted  onset  time  is  scored 
by  comparing  the  actual  onset  time  to  the  forecast  five  minutes  prior  to  the  event. 


To  score  predictions,  the  15-minute  period  following  the  prediction  is  searched.  If  no  event  occurs 
within  the  15  minutes  after  the  prediction,  the  prediction  is  false.  If  an  event  occurs  within  the  15  minutes 
after  the  prediction,  the  prediction  is  correct. 

The  performance  of  the  Terminal  Weather  Text  Message  was  assessed  for  the  Memphis  environment 
for  the  DemVal  (TABLE  19).  The  POP  was  0.96  and  the  PFP  was  0.33  for  moderate  precipitation.  For  heavy 
precipitation  the  POP  was  0.71  and  the  PFP  was  0.32 
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TABLE  19. 

Performance  of  the  Terminal  Weather  Text  Message  Precipitation  Predictions. 


1 

POP 

PFP  1 

1  Memphis  | 

Moderate  Precip 

0.96  (24/25) 

0.33  (96/289) 

Heavy  Precip 

0.71  (12/17) 

0.32  (46/142) 

Ail  Memphis 

0.86(36/42) 

0.33  (142/431) 

1  Orlando  | 

Moderate  Precip 

0.92  (22/24) 

0.41  (100/245) 

Heavy  Precip 

0.83  (10/12) 

0.30  (27/91) 

All  Orlando 

0.89  (32/36) 

0.38  (127/336) 

1  Both  Locations  | 

Moderate  Precip 

0.94  (46/49) 

0.37  (196/534) 

Heavy  Precip 

0.76  (22/29) 

0.31  (73/233) 

All 

0.87  (68/78) 

0.35  (269/767) 

The  performance  of  the  Terminal  Weather  Text  Message  was  assessed  for  the  Orlando  environment  for 
the  DemVal.  For  moderate  precipitation  events,  the  POP  was  0.92  and  the  PFT  was  0.41.  For  heavy 
precipitation,  the  POP  and  PFP  were  0.83  and  0.30,  respectively.  The  combined  Memphis-Orlando  perfor¬ 
mance  for  both  precipitation  intensities  was  a  POP  of  0.87  and  a  PFP  of  0.37. 


The  results  of  the  Memphis  demonstration  show  that  the  expected  precipitation  algorithm  works  well 
in  an  environment  with  fast  moving  storms  that  do  not  grow  or  decay  rapidly.  The  10-knot  storm  motion 
threshold  eliminated  many  potential  false  alarms  without  severely  impacting  the  POP  performance. 


In  Orlando,  the  causes  of  the  Expected  Moderate  Precipitation  false  alarms  are  two-fold:  storm  motion 
and  storm  decay.  The  Storm  Motion  algorithm  ran  only  on  heavy  precipitation  (VIP  level  3  or  greater)  and 
consequently  did  not  always  represent  the  motion  of  moderate  (VIP  level  2)  precipitation.  It  is  recommended 
that  the  Storm  Motion  algorithm  be  run  on  level  2  precipitation.  The  Expected  Moderate  Precipitation  mes¬ 
sage  could  then  be  generated  based  on  the  level  2  storm  motion. 
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In  addition,  because  storm  decay  information  was  not  available,  the  storms  sometimes  dissipated 
before  reaching  the  airport.  In  an  environment  such  as  Orlando,  attempting  to  predict  slow  moving  storms 
that  grow  and  dissipate  rapidly  is  difficult.  Without  the  ability  to  predict  storm  growth  and  decay,  the  perfor¬ 
mance  of  the  expected  precipitation  algorithm  will  be  degraded.  Training  materials  meed  to  emphasize  that 
“expected  precipitation”  is  a  simple  extrapolation  of  the  current  weather  and  does  not  account  for  storm 
evolution. 
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10.  CONCLUSIONS 


This  report  provided  an  algorithm-by-algorithm  performance  assessment.  For  each  product,  objective 
performance  criteria  were  defined.  The  minimum  performance  requirements  that  a  product  must  meet  to  be 
accepted  by  the  ATC  user  community  are  the  MOPR,  given  in  TABLE  1 .  These  are  the  requirements  by  which 
the  products  are  judged  acceptable  for  inclusion  into  IOC  ITWS. 


Truth  data  sets  were  created  for  each  product.  In  general,  these  data  are  generated  by  human  experts 
and  stored  in  machine-readable  format  to  support  automated  scoring.  Thus,  as  future  refinements  and  en¬ 
hancements  are  made  to  the  algorithms,  their  performance  can  rapidly  be  assessed. 


The  performance  assessment  provided  herein  has  shown  that: 


1 .  All  of  the  products  for  which  MOPR  are  stated  met  or  exceeded  those  requirements,  includ¬ 
ing  ITWS  Precipitation  (AP-edit),  Storm  Cell  Information,  Storm  Motion  and  Storm  Ex¬ 
trapolated  Position,  Microburst  Prediction,  Gust  Front  Forecast,  and  Terminal  Winds. 

2.  There  are  no  MOPR  listed  for  the  Lightning  product.  However,  the  lightning  data  were 
available  to  the  end-user  99.2  percent  of  the  time,  the  maximum  latency  of  the  warning  panel 
was  about  40  seconds,  and  less  than  0. 1  percent  of  the  lightning  reports  were  more  than  1 1 
nm  from  any  ASR-9  observed  weather  level  1  or  greater. 

3 .  There  are  no  MOPR  listed  for  the  Terminal  Text  Message  product.  The  performance  for  both 
moderate  and  heavy  precipitation  was  a  POP  of  0.87  and  a  PFP  of  0.37.  Analysis  indicates 
that  the  expected  precipitation  algorithm  works  well  in  an  environment  with  fast  moving 
storms  that  do  not  grow  or  decay  rapidly.  The  10-knot  storm  motion  threshold  eliminated 
many  potential  false  alarms  without  severely  impacting  the  POP  performance. 

4.  Training  for  operational  users  (e.g.,  Air  Traffic,  pilots,  airline  dispatchers)  will  need  to  ad¬ 
dress  performance  strengths  and  weaknesses  of  each  product.  This  was  identified  particular¬ 
ly  with  respect  to  the  SEP  and  Terminal  Text  as  “predictors”  of  future  precipitation  location. 

5 .  The  results  provided  herein  indicate  areas  where  the  performance  of  the  products  can  be  im¬ 
proved  relative  to  the  performance  goals  shown  in  TABLE  1: 

ITWS  Precipitation  (AP-edit)  has  exceeded  the  PEAP  goal.  The  performance  of  the  prod¬ 
ucts  relative  to  the  PEW  goal  has  yet  to  be  determined. 

Storm  Cell  Information  is  within  a  few  percent  of  meeting  its  goal  for  feature  association. 
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Microburst  Prediction  needs  work  on  the  “unrestricted”  mode  to  meet  its  operational  goals. 

Gust  Front  location  estimates  meet  the  goals  for  Memphis,  but  not  for  MCO.  Analysis  of 
data  from  different  climate  regimes  is  needed.  Although  the  number  of  cases  in  the  analysis 
is  limited,  the  wind  shift  estimate  does  not  meet  the  accuracy  goals. 

The  Terminal  Winds  product  meets  its  goals. 

SCI,  SEP,  and  Terminal  Weather  Text  Messages  products  need  to  account  for  storm  growth 
and/or  decay  to  meet  their  operational  goals. 
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APPENDIX  A. 

LIST  OF  ACRONYMS  AND  ABBREVIATIONS 


AGFS 

Aviation  Gridded  Forecast  System 

AP 

anomalous  propagation 

ARENA 

area  noted  for  attention 

ARP 

Airport  Reference  Point 

ARTCC 

Air  Route  Traffic  Control  Center 

ASR 

Airport  Surveillance  Radar 

ATC 

Air  Traffic  Control 

ATCT 

Airport  Traffic  Control  Tower 

CFP 

correct  forecast  probability 

comprefl 

composite  reflectivity 

DemVal 

Demonstration  and  Evaluation 

FAA 

Federal  Aviation  Administration 

FFP 

false  forecast  probability 

ITWS 

Integrated  Terminal  Weather  System 

LLWAS 

Low  Level  Windshear  Alert  System 

MBA 

microburst  alert 
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MDCRS  Meteorological  Data  Collection  and  Reporting  System 
MOPR  minimum  operational  performance  requirements 

NEXRAD  Next  Generation  Weather  Radar 

NLDL  National  Lightning  Data  Network 

NWS  National  Weather  Service 

ONERA  Office  Nationale  D-Etudes  et  de  Recherches  Aerospatiales 

PAWS  probability  of  accurate  wind  shift 

PEAP  probability  of  editing  AP 

PEW  probability  of  editing  weather 

PEWS  probability  of  erroneous  wind  shift 

PEA  probability  of  false  alarm 

PFD  percent  of  false  length  detected 

PFId  probability  of  generating  a  correct  forecast,  given  a  detection 

PFIe  probability  of  generating  a  correct  forecast,  given  an  event 

PFP  probability  of  false  prediction 

PLD  percent  of  length  detected 

POD  probability  of  detection 
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POP 

probability  of  correct  prediction 

POSH 

probability  of  severe  hail 

PWSA 

probability  of  generating  a  correct  wind  shear  alert 

PWSFA 

probability  that  wind  shear  alert  is  false 

SCI 

Storm  Cell  Information 

SEP 

Storm  Extrapolated  Position 

SSA 

Storm  Structures  Algorithm 

TDWR 

Terminal  Doppler  Weather  Radar 

TRACON 

Terminal  Radar  Approach  Control 

UT 

Coordinated  Universal  Time 

VIL 

vertically  integrated  liquid  water 

VIP 

Video  Integrator  and  Processor 

WSA 

wind  shear  alert 
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APPENDIX  B. 

DAY-BY-DAY  MICROBURST  PREDICTION  STATISTICS 


Table  B-1. 

Day-by-<Jay  performance  statistics  for  the  microburst  prediction  algorithm 

for  Memphis,  TN. 


DATE 

& 

TIME  (UT) 

POP 

PFP 

LEAD  TIME  (SECS) 

RES 

UNRES 

RES* 

UNRES 

RES** 

UNRES 

e-3-94 

1744-1840 

2100-2245 

1.00 

(2/2) 

1.00 

(2/2) 

0.00 

(0/2) 

0.20 

(12/61) 

282,96 

378,378 

6-4-94 

1910-2000 

— 

— 

0.00 

(0/19) 

— 

6-6-94 

2220-2259 

2315-0050 

0.50 

(1/2) 

0.50 

(1/2) 

0.00 

(0/2) 

0.00 

(0/5) 

117,- 

282,- 

6-7-«4 

0710-0810 

1015-1045 

0.00 

(0/1) 

0.00 

(0/1) 

0.00 

(0/1) 

1.00 

(49/49) 

6-0-94 

1722-1828 

— 

—  , 

— 

1.00 

(7/7) 

— 

— 

6-16-94 

2230-2315 

0.00 

(0/1) 

0.00 

(0/1) 

0.00 

(0/1) 

0.09 

(3/33) 

— 

— 

6-17-94 

2007-2051 

— 

— 

— 

— 

— 

6-21-94 

2155-2320 

— 

— 

— 

— 

— 

*  The  PFP  for  the  restricted  case  at  the  wind  shear  level  is  0.0  because  a  prediction  must  overlap 
a  wind  shear  event  to  be  issued. 

**  The  lead  time  for  each  microburst  is  reported  separately  in  this  column.  A  lead  time  of  0  indicates 
no  lead  time  and  a  dash  (-)  indicates  a  microburst  for  which  no  prediction  was  issued. 
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Table  B-1  continued. 


The  PFP  for  the  restricted  case  at  the  wind  shear  level  is  0.0  because  a  prediction  must  overlap 


a  wind  shear  event  to  be  issued. 

**  The  lead  time  for  each  microburst  is  reported  separately  in  this  column.  A  lead  time  of  0  indicates 
no  lead  time  and  a  dash  (-)  indicates  a  microburst  for  which  no  prediction  was  issued. 
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Table  B-1  continued. 


DATE 

& 

TIME  (UT) 

POP 

PFP 

LEAD  TIME  (SECS) 

RES 

UNRES 

RES* 

UNRES 

RES** 

UNRES 

7-11-94 

2227-2254 

— 

— 

— 

0.00 

(0/6) 

— 

TOTALS 

0.80 

(20/25) 

0.80 

(20/25) 

0.00 

(0/25) 

0.39 

(132/341) 

126 

246 

*  The  PFP  for  the  restricted  case  at  the  wind  shear  level  is  0.0  because  a  prediction  must  overlap 
a  wind  shear  event  to  be  issued. 


**  The  lead  time  for  each  microburst  is  reported  separately  in  this  column.  A  lead  time  of  0  indicates 
no  lead  time  and  a  dash  (-)  indicates  a  microburst  for  which  no  prediction  was  issued. 
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Table  B-2. 

Day-by-day  performance  statistics  for  the  microburst  prediction  algorithm 

for  Orlando,  FL. 


DATE 

& 

TIME  (UT) 

POP 

PFP 

LEAD  TIME  (SECS) 

RES 

UNRES 

RES* 

UNRES 

RES** 

UNRES 

7-14-94 

0.80 

0.80 

0.00 

0.13 

290,114 

643,177 

2000-2200 

(4/5) 

(4/5) 

(0/5) 

(14/105) 

114,0 

467,527 

7-16-94 

2012-2120 

0.50 

(1/2) 

0.50 

(1/2) 

0.00 

(0/2) 

73,- 

73,- 

7-21-94 

0.83 

1.00 

0.00 

0.03 

0,0 

578,0 

1504-1739 

(5/6) 

(6/6) 

(0/6) 

(1/40) 

41,114 

237,- 

104,177 

237,289 

7-27-94 

1936-2049 

1.00 

(1/1) 

1.00 

(1/1) 

0.00 

(0/1) 

0.20 

(3/15) 

115 

353 

7-28-94 

1819-1936 

0.00 

(0/3) 

0.33 

(1/3) 

0.00 

(0/3) 

0.42 

(31/74) 

f  t 

-.248,- 

7-30-94 

2032  -2200 

0.50 

(1/2) 

0.50 

(1/2) 

0.00 

(0/2) 

0.55 

(12/22) 

0,- 

0,- 

8-9-94 

1929-2139 

0.00 

(0/1) 

1.00 

(1/1) 

0.00 

(0/1) 

0.00 

(0/4) 

— 

538 

8-15-94 

0.40 

0.40 

0.00 

0.03 

185,174 

185,527 

1952-2114 

(2/5) 

(2/5) 

(0/5) 

(1/39) 

1  1 

TOTALS 

0.56 

(14/25) 

0.68 

(17/25) 

0.00 

(0/25) 

0.19 

(63/329) 

58 

205 

1 

*  The  PFP  for  the  restricted  case  at  the  wind  shear  level  is  0.0  because  a  prediction  must  overlap 
a  wind  shear  event  to  be  issued. 

**  The  lead  time  for  each  microburst  is  reported  separately  in  this  column.  A  lead  time  of  0  indicates 
no  lead  time  and  a  dash  (-)  indicates  a  microburst  for  which  no  prediction  was  issued. 
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APPENDIX  C 

MEMPHIS  GUST  FRONT  WIND  SHIFT  AND  WIND  SHEAR  DATA 
USED  FOR  PERFORMANCE  EVALUATION 


WIND  SHIFT 

WIND  SHEAR  | 

MIGFA 

LLWAS  TRUTH 

TDWR  TRUTH 

MIGFA 

TRUTH 

Date 

Time 

(GMT) 

Dir 

Speed 

(m/s) 

Dir 

Speed 

(m/s) 

Dir 

Speed 

(m/s) 

DV 

(m/s) 

DV 

(m/s) 

06/05/94 

2005 

183.5 

4.2 

198.3 

■B 

200 

4.0 

■B 

2.0 

06/07/94 

0339 

330.6 

6.4 

0.2 

4.7 

355 

6.0 

11.4 

06/07/94 

0523 

332.3 

294.4 

6.5 

310 

9.0 

6.9 

7.4 

06/07/94 

0805 

278.5 

4.9 

313.8 

6.2 

350 

6.0 

8.0 

9.0 

06/09/94 

1758 

283.1 

21.0 

280.0 

18.4 

300 

24.0 

9.5 

13.8 

06/16/94 

2123 

85.1 

6.1 

140.0 

6.2 

160 

6.5 

4.3 

5.0 

06/22/94 

2034 

140.7 

mm 

163.3 

10.6 

180 

10.0 

6.9 

9.0 

06/28/94 

1420 

343.7 

14.6 

47.1 

4.2 

20 

5.0 

10.7 

12.0 

07/01/94 

0244 

283.7 

8.0 

248.8 

2.5 

300 

1.0 

12.3 

12.0 

07/03/94 

2346 

134.9 

6.0 

160.0 

5.8 

160 

6.0 

3.3 

4.2 

07/04/94 

2310 

52.4 

3.5 

320.0 

1.0 

330 

2.0 

5.3 

4.5 

07/05/94 

1959 

54.0 

2.9 

176.7 

2.0 

40 

1.0 

5.2 

4.5 

07/06/94 

2226 

48.8 

5.3 

67.6 

6.6 

40 

8.0 

6.5 

07/08/94 

1337 

311.1 

5.3 

291.0 

mm 

290 

6.0 

7.0 

6.5 

07/08/94 

1850 

289.1 

6.1 

257.9 

5.3 

300 

4.5 

10.5 

11.0 

07/08/94 

1835 

287.2 

3.9 

253.1 

300 

4.0 

5.1 

2.0 

07/08/94 

2148 

136.3 

160.0 

H3I 

160 

8.0 

4.6 

2.8 
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